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The  report’s  summary,  reviews  the  major  finding  and  recommendations  of  the  National 
Research  Council  (NRC)  Committee  on  Air  Force  Research  Management.  The  committee  was 
formed  at  the  request  of  the  Air  Force  Systems  Command  to  assess  the  effectiveness  of  Air 
Force  management  of  basic  (6.1)  research  and  to  recommend  any  needed  improvement. 
Subcommittee  reports,  on  which  the  summary  is  based,  are  included  as  appendices. 

The  context  of  defense  basic  research  has  changed  greatly  in  recent  years.  With  the  end 
of  the  Cold  War  this  century’s  principal  threat  to  national  security  has  receded,  and  the  threats, 
more  localized  but  more  unpredictable,  have  emerged.  The  nation  also  is  facing  strong 
budgetary  pressures  requiring  that  many  national  endeavors  be  rethought.  Moreover,  defense- 
related  basic  research  itself  has  changed.  It  is  now  increasingly  a  global  enterprise  connected 
outside  the  U.S.  services  and  beyond  U.S.  borders.  For  these  and  other  reasons,  many  recent 
reports  have  focused  on  the  problems  and  needs  of  defense  research  management  (and  defense 
management  more  generally).  All  these  factors  played  a  role  in  prompting  Air  Force  interest 
in  the  issues  of  basic  research  management  which  are  addressed  in  this  report. 
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Summary  Report 


This  summary  report  reviews  major  findings  and  recommendations  of  the 
National  Research  Council  (NRC)  Committee  on  Air  Force  Research 
Management.  The  committee  was  formed  under  the  NRC’s  Air  Force  Studies 
Board,  at  the  request  of  Air  Force  Systems  Command,  in  the  spring  of  1991.^ 
The  object  of  this  study  was  to  assess  the  effectiveness  of  Air  Force 
management  of  basic  (6.1)  research  and  to  recommend  any  needed 
improvements. 

The  context  of  defense  basic  research  has  changed  greatly  in  recent 
years.  With  the  end  of  the  Cold  War,  the  principal  threat  to  national  security 
of  this  century  has  receded,  and  other  threats,  more  localized  but  more 
unpredictable,  have  emerged.  The  nation  also  is  facing  strong  budgetary 
pressures,  requiring  that  many  national  endeavors  be  rethought.  Moreover, 
defense-related  basic  research  itself  has  changed.  It  is  now  increasingly  a 
global  enterprise,  conducted  outside  the  U.S.  services  and  beyond  U.S. 
borders.  For  these  and  other  reasons,  many  recent  reports  have  focused  on 
the  problems  £md  needs  of  defense  research  management  (and  defense 
management  more  generally).  All  these  factors  played  a  role  in  prompting  the 
Air  Force  interest  in  the  issues  of  basic  research  management  addressed  here. 

These  trends  also  underlie  recent  organizational  changes  in  the  services. 
All  three  services  have  been  engaged  in  consolidating  and  realigning  their 
laboratories  and  reorganizing  their  research-related  processes.  In  the  Air 
Force,  the  original  study  sponsor.  Air  Force  Systems  Command  (AFSC),  was 
combined  with  Air  Force  Logistics  Command,  to  form  the  new  Air  Force 
Materiel  Command  (AFMC),  as  of  July  1,  1992,  to  unite  research  and 
development  and  logistics  with  the  goal  of  new  eflSciencies.^  The  Air  Force 


^  Because  this  study  relied  on  the  use  of  three  subcommittees  in  approaching  its  task,  this 
summary  report  serves  both  as  a  consensus  report  (Appendixes  A-C  provide  the  three 
subcommittee  reports)  and  as  an  executive  summary. 

^  These  areas  had  been  united  before,  under  Air  Materiel  Command  (and  its  predecessor, 
Air  Corps  Materiel  Division),  but  they  were  functionally  divided  during  World  War  11,  and 
broken  out  under  separate  organizations  in  1950. 
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Office  of  Scientific  Research  (AFOSR),  which  is  the  Air  Force  organization 
directly  responsible  for  most  Air  Force  basic  research  (in  particular,  the 
Defense  Research  Sciences  program),^  was  once  an  element  of  AFSC;  it  is 
now  within  AFMC. 

Insofar  as  possible,  the  committee  has  attempted  to  develop  its 
observations  and  recommendations  with  attention  to  all  these  developments 
in  the  environment  for  Air  Force  basic  research  management. 

The  original  study  charges  to  the  committee  are  as  follows: 

1.  Compare  the  level,  management,  and  allocation  procedures  for  the 
6.1  resources  of  each  of  the  three  services. 

2.  Assess  the  role  and  interaction  of  AFSC  [AFMC]  laboratories  with 
AFOSR  in  the  conduct  of  6.1  research. 

3.  Evaluate  the  quality  of  the  research  products  of  AFOSR  and  the 
products  of  6.1  in-house  research. 

4.  Assess  the  means  and  extent  of  technology  transition  by  AFOSR  and 
by  performers  of  6.1  research. 

5.  Determine  the  extent  of  coverage  by  AFOSR  of  technical  areas  that 
have  strong  research  activity  by  industry. 

6.  Identify  technical  areas  that  need  greater  coverage  and  areas  that 
receive  more  attention  than  the  research  value  warrants. 

7.  Compare,  to  a  reasonable  degree,  AFOSR  research  and  that  of  other 
research  organizations. 

8.  Recommend  ways  to  improve  Air  Force  research  management. 

In  initial  meetings  of  the  committee,  it  became  clear  that  the  answers  to 
these  charges  woiild  overlap  in  critical  ways,  and  that  the  fifth  and  sixth 
charges— concerning  the  leveraging  of  research,  satisfaction  of  critical  research 
needs,  and  avoidance  of  research  redundancy— could  be  addressed  within  the 
scope  of  this  study  only  through  a  focus  on  management  processes.  A  focus 
on  process  also  allowed  the  committee  to  offer  its  insight  on  valuable  general 
methods,  rather  than  simply  scattered,  specific  observations  about  current  Air 
Force  programs. 

To  best  respond  to  its  charge  from  the  AFSC  Commander,  the 
committee  thus  formed  three  working  groups,  with  the  following  emphases: 

•  interactions  between  AFOSR  and  the  Air  Force  laboratories 
(Subcommittee  A); 


^  Defense  Research  Sciences  is  the  program  element  6.1102F  in  the  Department  of  Defense 
budget,  representing  most  U.S.  defense  basic  research. 
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•  comparison  of  research  management  practices  at  AFOSR  and  other 
DOD  and  Federal  Agencies  (Subcommittee  B);  and 

•  quality  of  AFOSR-sponsored  research  (Subcommittee  C). 

While  the  main  focus  of  Subcommittee  A’s  study  was  the  second  charge, 
of  Subcommittee  B’s  the  seventh,  and  of  Subcommittee  Cs  the  third,  all  three 
subcommittees  also  considered  all  the  remaining  original  charges,  within  their 
respective  scopes.  Much  of  the  committee’s  information-gathering  and 
deliberation  took  place  within  the  three  subcommittees,  and  all  three 
subcommittees  prepared  written  reports  (Appendixes  A-C),  on  which  this 
siimmary  report  is  based. 

In  the  following  review  of  findings  and  recommendations  (organized  by 
the  original  list  of  charges  above),  the  views  presented  represent  the 
consensus  of  the  full  committee. 

In  carrying  out  its  study,  the  full  committee  or  its  subcommittees  were 
briefed  by  the  AFSC  Director  of  Science  and  Technology,  the  AFSC  Chief 
Scientist,  the  AFOSR  Director,  and  senior  managers  and  representatives  of 
AFOSR,  the  Air  Force  laboratories.  Army  Research  Organization,  Office  of 
Naval  Research,  Office  of  the  Secretary  of  Defense,  Defense  Advanced 
Research  Projects  Agency,  Air  Force  Scientific  Advisory  Board,  National 
Science  Foundation,  U.S.  Department  of  Energy,  and  National  Aeronautics 
and  Space  Administration  (see  Appendix  D  fora  list  of  these  meetings). 
Committee  members  also  made  site  visits  to  all  four  of  the  Air  Force’s  newly 
formed  "super"  laboratories:  the  Armstrong,  Phillips,  Rome,  and  Wright 
laboratories. 

Based  on  the  information  provided,  the  committee  reached  the  following 
major  findings  and  recommendations,  for  each  of  the  original  study  charges. 
The  reader  is  encouraged  to  consult  the  subconunittee  reports  in  the 
appendices,  for  background  information  and  analyses  provided  by  the  working 
groups. 


LEVEL,  MANAGEMENT,  AND  ALLOCATION  PROCEDURES 
FOR  6.1  RESOURCES  OF  THE  THREE  SERVICES 

DOD  funding  for  basic  research  has  been  reasonably  stable  in  constant 
dollars  over  the  past  decade.  Stability  of  funding  is  critical  for  basic  research, 
because  its  important  results  are  generally  less  predictable  and  seen  over  a 
longer  term  than  in  applied  research.  Briefers  of  the  committee  indicated  that 
the  Air  Force  is  attempting  to  maintain  the  stability  of  its  basic  research 
funding,  even  in  the  face  of  likely  severe  budget  cuts  over  the  next  several 
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years.  This  goal  of  funding  stability  is  both  desirable  and  practical,  if  the 
benefits  of  basic  research  are  to  be  achieved. 

According  to  information  provided  by  DOD  and  the  services’  offices  of 
research,  Army  6.1  funding  dming  fiscal  year  1992  is  approximately  $197 
million,  Navy  6.1  funding  $422  million,  and  Air  Force  6.1  funding  $203  million 
(see  Appendix  B,  Figure  B-4).  These  values  represent  3.6  percent  of  the 
Army’s  total  Research,  Development,  Test  and  Evaluation  (RDT&E)  funding, 
4.2  percent  of  the  Navy’s;  and  1.5  percent  of  the  Air  Force’s  (Figure  B-5). 
Thus,  the  Air  Force  appears  to  commit  a  significantly  smaller  percentage  of 
its  total  RDT&E  funds  to  basic  research  than  either  of  the  other  two  services. 
The  reasons  for  this  discrepancy  (and  whether  or  not  the  ratios  reflect 
realities)  were  not  made  clear,  and  the  committee  was  unable  to  investigate 
this  complex  subject  in  satisfactory  depth.  The  Air  Force  should  consider 
whether  further  study  of  this  issue  is  in  order. 

With  the  formation  of  AFMC,  AFOSR  has  been  placed  within  a  much 
larger,  more  diverse  organization.  While  some  of  the  likely  benefits  of  the 
merger  are  clear,  the  committee  and  other  experts  are  concerned  that,  under 
the  new  organization,  AFOSR  may  be  too  far  removed  from  senior 
leadership,  and  thus  basic  research  may  be  too  far  removed  from  planning 
and  policy.  Geopolitical  and  budgetary  trends  have  changed  the  defense 
environment,  but  basic  research  remains  as  critical  to  achieving  the  Air  Force 
mission  as  ever—it  may  even  be  more  critical  today,  given  the  worldwide 
diffusion  of  high  technology  and  new  competition  in  research.  Such 
observations  raise  the  concern  that  basic  research  may  not  be  given  adequate 
emphasis  within  the  new  Air  Force  structure. 

The  services  distribute  their  basic  research  funds  through  different 
organizational  arrangements.  The  Army  Research  Office  manages  only  the 
Army’s  extramural  programs  and  its  Centers  of  Excellence,  that  is,  primarily 
university  research,  representing  about  one-third  of  the  Army’s  basic  research 
budget.  The  Office  of  Naval  Research  manages  all  Navy  6.1  funding.  AFOSR 
also  manages  all  Air  Force  6.1  funds,  except  for  In-House  Laboratory 
Independent  Research  (HJR)  funds.  There  are  advantages  and  disadvantages 
of  these  different  practices,  but  no  information  provided  to  the  committee 
suggested  that  Air  Force  practice  should  be  changed.  That  is,  the  committee 
saw  no  evidence  to  question  AFOSR’s  designation  as  the  "single  manager"  of 
the  Defense  Research  Sciences  program. 

Current  AFOSR  guidance  suggests  that  about  one-third  of  its  Defense 
Research  Sciences  program  funding  should  go  to  intramural  research."* 
In-house  basic  research  programs  are  essential,  for  mission-specific  work,  and 


^  "Guiding  Prindples  for  the  Intramural  Basic  Research  Program  Sponsored  by  AFOSR" 
(see  Appendix  A,  Attachment  I,  of  this  report). 
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perhaps  more  critically,  to  absorb  the  advances  of  the  outside  research 
community.  Precisely  because  of  the  extensive  ej^ertise  in  the  outside 
research  community,  Air  Force  support  of  extramural  basic  research  is  at  least 
equally  vital.  Moreover,  extramural  research  offers  special  benefits  beyond 
connection  with  the  outside  research  community  and  the  opportunity  to  shape 
its  interests.  Air  Force  extramural  research  is  highly  leveraged,  in  funds, 
equipment,  and  intellectual  resources;  it  also  prepares  the  Air  Force’s  future 
science  and  engineering  work  force,  notably  through  the  extensive  involvement 
of  graduate  students  in  university  research  laboratories.  While  current 
budgetary  trends  may  increase  pressures  within  the  Air  Force  to  allocate  a 
greater  percentage  of  basic  research  dollars  in  house,  the  above  considerations 
as  a  whole  suggest  that  this  would  not  be  a  prudent  policy  shift.  (Once  the 
unique  contributions  of  intra-  and  extramural  programs  have  been  factored 
into  allocation  decisions,  the  only  criteria  the  Air  Force  should  use  in  its 
funding  decisions  are  research  relevance,  quality,  and  productivity.) 


Recommendations 

1.  The  Air  Force  should  maintain  its  6.1  funding  at  no  less  than  the 
current  level,  adjusted  for  inflation  (and  should  perhaps  consider  devoting  a 
greater  percentage  of  RDT&E  funding  to  basic  research). 

2.  The  importance  of  basic  research  to  the  Air  Force’s  long-term 
warfighting  capabilities  must  be  recognized  by  its  organizational  structure. 
Given  its  current  structure,  the  Air  Force  might  best  achieve  this  goal  by 
ensuring  that  the  Air  Force  Technology  Executive  Officer  continues  to  be 
assigned  his  current  level  of  responsibility  and  role,  and,  by  virtue  of  training 
and  background,  continues  to  demonstrate  strong  knowledge  and  appreciation 
of  basic  research. 

3.  The  Air  Force  should  not  make  any  policy  change  to  shift  basic 
research  funding  from  extramural  to  in-house  programs. 


THE  ROLE  AND  INTERACTION  OF  AFMC  LABORATORIES 
WITH  AFOSR  IN  THE  CONDUCT  OF  6.1  RESEARCH 

A  number  of  strategic  planning  processes  shape  the  development  of  Air 
Force  basic  research  programs:  formal  guidance  received  fi'om  the  Assistant 
Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ)  and  Director  of 
Defense  Research  and  Engineering,  preparation  of  the  Air  Force  Research 
Technology  Area  Plan  (TAP),  and  Tri-Service  S&T  Reliance  documents  along 
with  five  complementary  cross-cutting  interlaboratory  investment  plans. 
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However,  in  the  conunittee’s  view,  none  of  these  processes,  individually  or 
together,  provides  an  adequate,  integrated  strategic  plan  for  basic  research 
that  is  oriented  to  meeting  Air  Force  mission  requirements. 

Good  mission-specific  planning— which  receives  high-level  attention— will 
be  increasingly  vital  in  an  era  of  tight  budgets.  For  example,  a  clear,  directed 
plan  of  basic  research  will  be  needed  to  ensure  the  effectiveness  and 
survivability  of  aircraft  that  must  remain  in  operation  for  many  years.  These 
and  other  critical  Air  Force  needs  must  be  met,  beginning  with  strategic 
planning.  The  committee  saw  laudable  efforts  in  this  direction  on  the  part  of 
AFOSR,  which  has  emphasized  research  in  a  number  of  these  particularly 
critical  areas. 

Mission-oriented  planning  should  reflect  the  judgment  of  AFOSR,  the 
laboratories.  Air  Logistics  Centers,  operational  commands,  and  representatives 
of  the  larger  R&D  community.  (One  approach  that  has  been  pplied  to 
technology  planning,  the  "Strategy  to  Task"  approach,  in  which  operators  and 
researchers  are  involved  in  an  active  exchange  of  ideas,  may  also  be  of  value 
in  planning  basic  research.) 

Strategic  research  planning  in  the  new  environment  will  require  an 
increased  focus  on  Air  Force  mission  needs,  but  it  should  be  emphasi-  ;d  that 
this  focus  should  not  lead  to  using  6.1  funds  for  applied  research.  Basic 
research  can  be  more  oriented  to  mission  needs  and  still  be  fundamental 
science  (e.g.,  basic  science  relating  to  corrosion).  Moreover,  to  achieve  the 
revolutionary  advances  of  the  future,  some  basic  research  must  be  strictly 
science-driven,  pursued  without  concern  for  application. 

The  formication  of  specific  research  programs  is  now  largely  a  top-down 
process.  Although  AFOSR  management  staff  receive  guidance  from  a  number 
of  sources,  they  are  largely  responsible  for  funding  decisions,  initiation  and 
guidance  of  research,  and  program  quality.  They  are  assisted  in  program 
planning  by  several  advisory  bodies,  such  as  the  AFOSR  Research  Council, 
which  represents  the  laboratories  through  the  participation  of  their  chief 
scientists,  and  the  Air  Force  Scientific  Advisory  Board. 

Nevertheless,  while  valuable,  these  arrangements  do  not  sufficiently 
involve  those  beyond  AFOSR  staff  in  program  planning.  Program  planning 
would  benefit  from  the  greater  involvement  of  laboratory  staff,  because  of 
their  special  knowledge  of  Air  Force  technical  requirements  and  their  ability 
to  better  facilitate  the  transition  of  research  results  when  they  are  familiar 
with  the  research.  Laboratory  staff  report  they  are  dissatisfied  with  their 
current  level  of  involvement  in  program  planning.  However,  in  involving 
laboratory  staff  in  program  planning,  care  should  be  taken  to  avoid  conflicts 
of  interest  (because  the  laboratories  compete  with  extramural  programs  for 
basic  research  funding  from  AFOSR)  and  to  avoid  the  diffusion  of  leadership 
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in  program  planning,  which  could  lead  to  poor  coordination  of  research  or 
tolerance  of  weak  components. 

The  greater  involvement  in  program  planning  of  external  peers  from 
both  academia  and  industry  could  also  be  highly  beneficial.  Currently,  AFOSR 
research  managers  vary  greatly  in  the  way  they  seek  external  views  on 
program  formulation  and  funding  decisions.  For  some  AFOSR  areas,  such  as 
Life  and  Enviromnental  Sciences  and  Chemical  Sciences,  external  peer  review 
of  proposals  is  a  formalized  process.  In  other  cases,  the  external  review  sought 
is  simply  ad  hoc.  The  more  casual  approaches  raise  concerns.  External 
perspective  on  Air  Force  research  programs  can  be  invaluable  in  keeping 
programs  at  the  leading  edge  of  research,  leveraging  the  work  of  others, 
avoiding  research  redundancy,  introducing  appropriate  multidisciplinary 
thrusts,  and  achieving  the  highest  quality  research  programs  in  general. 

At  the  same  time,  like  laboratory  input  to  program  planning,  external 
review  can  present  its  own  problems,  including  burdens  of  time,  expense,  and 
administrative  work,  and  the  possible  introduction  of  other  kinds  of  biases 
(see  Appendixes  A  and  B  for  background  discussion). 

Considered  together,  the  benefits  and  drawbacks  of  involving  external 
peers  in  program  formulation  lead  committee  members  to  hold  a  range  of 
views.  Some  believe  that  external  proposal  review  should  be  formalized  across 
AFOSR,  others  that  program  managers  should  be  encouraged  (e.g.,  through 
performance  reviews)  to  pursue  greater  external  review,  but  in  their  own  ways. 
Yet  others  would  argue  for  convening  outside  advisory  groups  to  review 
program  plans  in  specific  discipline  areas,  or  for  annual  AFOSR  surveys  of 
outside  research.  However,  the  committee  as  a  whole  agrees  strongly  that 
external  input  to  program  planning  should  be  given  greater  emphasis,  for  all 
the  reasons  above. 

The  centralization  of  authority  in  Air  Force  basic  research  planning  also 
increases  the  risk  that  good  innovative  research  may  go  unfunded.  If  proposed 
research  is  not  of  direct  interest  to  a  program  manager,  support  for  that 
research  is  likely  to  be  minimal.  AFOSR  has  not  always  moved  well  into  new 
areas  of  research,  even  when  given  the  guidance  to  do  so  (e.g.,  at  SAF/AQ’s 
request  several  years  ago  that  it  develop  stronger  research  in  logistics). 
AFOSR  should  be  pursuing  strong  programs  of  basic  research  in  areas  that 
bear  on  logistics  and  manufactming  as  well  (e.g.,  in  corrosion,  stress-cracking, 
tribology,  artificial  intelligence  for  monitoring  systems,  and  nondestructive 
evaluation). 

One  traditional  research  management  problem  is  balancing  nearer-  and 
longer-term  goals.  The  committee  heard  reports  both  that  proposed  basic 
engineering  research  is  sometimes  dismissed  as  "too  applied"  and  that 
pressures  within  the  Air  Force  sometimes  lead  in-house  labs  to  overemphasize 
near-term  results  in  their  basic  research,  in  this  case  forcing  the  choice  of 
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projects  properly  called  "applied."  Policies  and  procedures  can  be  designed  to 
help  avoid  such  problems.  Ratings  of  "relevancy,"  for  example,  should  be 
clearly  distinguished  from  the  idea  of  "promising  near-term  results."  Such 
issues  could  be  further  considered  through  a  regular  seminar  for  program 
managers  (other  likely  seminar  topics  would  include  techniques  to  manage 
both  small  research  groups  and  large  integrated  teams,  and  case  histories  in 
transition;  see  Appendix  A). 


Recommendations 

1.  Especially  in  view  of  current  pressures  on  defense  budgets,  strategic 
planning  for  basic  research  should  take  greater  account  explicitly  of  long-term 
Air  Force  mission  goals.  This  planning  should  reflect  the  judgment  of  AFOSR, 
the  laboratories.  Air  Logistics  Centers,  operational  commands,  and 
representatives  of  the  larger  R&D  community. 

2.  A  small  portion  of  Air  Force  basic  research  funding  should  also  be 
set  aside  to  pursue  especially  innovative  research,  regardless  of  the 
applications  it  promises. 

3.  The  practice  of  briefing  the  basic  research  strategic  plan  to  senior 
Air  Force  leaders  at  AFMC  and  Air  Force  Headquarters  is  strongly  endorsed, 
along  with  other  measures  that  enhance  high-level  understanding  of  the 
potential  roles  of  basic  research  in  meeting  Air  Force  mission  requirements. 

4.  To  improve  the  formulation  of  basic  research  programs,  AFOSR 
should  involve  at  more  of  a  "partnership"  level  those  beyond  its  own  staff, 
both  those  in  the  laboratories  and  external  experts.  Carefully  designed 
procedures  will  be  needed  for  this  enhanced  involvement  to  be  successfully 
achieved. 

5.  In  improving  program  planning  procedures,  the  Air  Force  should 
consider  better  ways  to  incorporate  critical  new  areas  of  research,  for 
example,  by  establishing  an  interdisciplinary  team  of  program  managers, 
augmented  appropriately  by  outside  experts,  to  evaluate  proposed  research  in 
new  and  interdisdplinaiy  areas. 

6.  Air  Force  research  management  should  develop  policies  and 
procedures  to  help  program  managers  avoid  arbitrary  decisions  in  determining 
the  ’Tjasic"  or  "developmental"  nature  of  proposed  research.  For  example,  the 
nature  of  "relevan<y"  ratings  should  be  made  clear.  Additionally,  a  continuing 
education  seminar  for  program  managers  could  be  established  to  address  such 
management  questions. 
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QUALITY  OF  THE  RESEARCH  PRODUCTS  OF  AFOSR 
AND  OF  6.1  IN-HOUSE  RESEARCH 

AFOSR  has  recently  improved  its  procedures  to  assess  research  quality. 
"Relevancy"  reviews  conducted  at  each  of  the  Air  Force  laboratories  during 
the  fall  permit  the  labs  to  evaluate  the  AFOSR  basic  research  programs  in 
general,  with  emphasis  on  their  responsiveness  to  Air  Force  needs,  "Quality" 
reviews  are  also  held  at  the  fall  meetings,  in  which  invited  members  of  the  Air 
Force  Scientific  Advisory  Board  and  AFOSR  scientific  directors  complete 
quality  review  forms.  Two  overall  ratings,  one  for  quality  and  one  for 
relevance,  are  thus  produced  for  each  program. 

These  procedures  represent  a  significant  advance  over  previous  methods; 
however,  they  still  have  shortcomings.  Because  of  the  many  work  units 
covered  in  the  current  reviews,  the  reviews  can  be  only  cursory.  Attendance 
of  external  peer  reviewers  is  also  quite  limited.  The  review  of  basic  research 
programs  is  therefore  largely  internal,  and  in-depth  review  of  program  quality 
remains  highly  dependent  on  the  individual  program  manager.  While  many 
AFOSR  program  managers  are  clearly  very  talented  and  capable,  this 
dependence  on  so  few  experts  for  the  review  of  research  is  a  limiting 
management  strategy  in  today’s  research  environment.  Research  is  now  far 
more  widely  conducted  than  in  the  past,  and  embodies  many  more  efforts  and 
specialized  fields  than  can  be  mastered  by  so  few  individuals. 

The  available  evidence  suggests  that  the  Air  Force  has  a  sound  program 
of  basic  research.  Informal  review  of  AFOSR  projects  led  the  committee  to 
conclude  that  the  quality  of  many  is  high.  AFOSR  has  also  recently  instituted 
the  Star  Team  Award,  to  recognize  its  leading  Air  Force  laboratory  research 
teams.  These  awards  are  based  on  well-specified  criteria,  and  they  should  be 
a  valuable  tool  in  promoting  high-caliber  research. 

At  the  same  time,  serious  difficulties  were  encountered  in  attempting  to 
assess  the  overall  quality  of  AFOSR-sponsored  research,  whether  intra-  or 
extramural,  or  as  a  whole.  No  systematic  quality  metrics  are  maintained  on 
Air  Force  basic  research.  Such  quality  metrics  are  admittedly  difficult  to 
specify.  However,  a  number  of  established  indicators,  while  imperfect,  could 
provide  some  objective  benchmarks  of  research  quality.  Their  tystematic 
application  would  be  of  help  in  assessing,  improving,  and  justifying  the  value 
of  Air  Force  research  programs. 

The  committee  therefore  developed  a  list  of  such  quality  indicators  of 
special  interest  to  AFOSR  (Appendix  C).  For  research  personnel,  these 
objective  quality  metrics  include  fellowships  in  professional  societies, 
memberships  in  national  academies,  and  other  forms  of  professional 
recognition,  such  as  society  prizes  and  awards.  Additional  useful  metrics  are 
total  research  personnel  breakdown  by  degrees  and  disciplines,  and  average 
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professional  age  (and  age  distribution  by  fields  and  degrees).  To  benchmark 
the  quality  of  research  results,  useful  metrics  include  publications  in  refereed 
joiumals,  citations,  and  patent  activity.  The  systematic  and  regular  capture  of 
all  such  data  could  be  used  to  evaluate  and  improve  the  quality  of  the 
AFOSR  researcher  pool  and  the  quality  of  AFOSR  research  results. 

The  collection  of  several  additional  kinds  of  information  could  also  be 
of  great  value  to  AFOSR.  Ultimately,  the  transition  of  research  results  is  one 
of  the  most  important  criteria  of  research  quality  and  relevancy  from  the 
standpoint  of  AFOSR.  While  much  anecdotal  information  is  available  about 
the  transition  of  Air  Force-sponsored  research,  systematic  information  is  not 
kept.  For  this  reason,  the  Air  Force’s  success  in  transition  of  research  results 
cannot  be  well  evaluated,  and  the  management  steps  that  might  improve  this 
transition  are  hard  to  identify.  Similarly,  information  on  project  results  (for 
both  successful  and  unsuccessful  projects)  could  help  improve  project 
management  and  justification  throughout  the  R&D  management  chain. 
Project  results  are  now  documented  in  various  ways,  but  there  is  no  systematic 
capture,  storage,  and  distribution  of  such  information,  although  the  committee 
saw  indications  that  the  AFOSR  Director  is  instituting  efforts  in  this  direction. 

The  quality  of  Air  Force  basic  research  also  depends  necessarily  on  the 
quality  of  the  science  and  engineering  work  force  over  the  long  term.  This 
work  force  includes  not  only  research  personnel  in  Air  Force  laboratories,  but 
also  researchers  in  universities  and  industry  who  directly  and  indirectly 
support  Air  Force  RDT&E.  To  secure  the  reliable  delivery  of  technology  over 
time,  AFOSR  has  been  active  in  ensuring  that  such  long-term  work  force 
needs  are  met.  AFOSR  programs  for  postdoctoral  and  faculty  exchange,  and 
for  science  and  engineering  education,  are  all  aimed  at  supporting  and 
developing  this  larger  Air  Force  work  force. 

Tfre  study  subcommittee  that  reviewed  these  current  Air  Force  programs 
concluded  that  they  provide  financial  support  for  a  substantial  number  of 
faculty  and  postgraduate  and  graduate  students  (Appendix  C).  AFOSR  science 
anu  engineering  education  programs  also  appear  well  designed  to  meet  their 
objectives.  However,  especially  in  this  period  of  tightening  defense  budgets, 
the  committee  is  concerned  that  critical  funding  support  in  these  areas  may 
be  cut,  while  no  quantitative  assessment  has  been  made  of  future  Air  Force 
scientiiSc  work  force  needs.  A  good  assessment  should  take  into  account  the 
leveraging  and  technology  transfer  opportunities  these  forms  of  support 
provide.  Additionally,  it  should  be  noted  that  significant  time  is  required  to 
bring  research  personnel  through  the  educational  pipeline,  and  it  would  be 
wise  for  the  Air  Force  to  make  conservative  assessments  in  this  light.  (Such 
strategic  planning  must  be  based  on  incomplete  facts,  but  the  planmng  process 
forces  a  valuable  discipline,  yielding  important  insights;  and  plans  will  be 
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amended  both  with  this  experience  and  the  availability  of  more  accurate 
information.) 


Recommendations 

1.  To  improve  its  oversight  of  research  quality,  AFOSR  should 
randomly  select  projects  representing  some  minimum  percentage  of  its  aimual 
basic  research  outlay  (at  least  10  percent)  and  subject  these  projects  to 
detailed  external  peer  review.  Such  reviews  would  provide  valuable  quality 
information  and  help  validate  internal  reviews. 

2.  AFOSR  should  conduct  internal,  in-depth  reviews  of  one-third  of  its 
research  projects  each  year,  so  that  every  research  project  with  a  life  of  three 
years  or  more  is  subjected  to  rigorous  benchmarking. 

3.  An  organized  database  should  be  created  and  maintained  on 
objective  quality  metrics  for  AFOSR  researchers,  and  should  be  used  as  a  tool 
to  improve  the  overall  quality  of  the  pool  of  researchers. 

4.  Organized  databases  should  also  be  created  and  maintained  on 
publications  in  refereed  journals,  citations,  and  patent  activity,  to  help  assess 
and  improve  the  quality  of  research  results  (including,  e.g.,  to  judge  the 
relative  contributions  of  intra-  and  extramural  programs). 

5.  AFOSR  should  institute  a  process  to  capture  information  on  the 
scientific  impact  and  transition  of  its  research  results,  to  measure  the 
effectiveness  of  its  programs,  improve  its  portfolio,  and  justify  its  research 
programs.  (For  more  detail,  see  the  section  that  follows.) 

6.  A  systematically  organized  knowledge  base  of  lessons  learned  from 
research  projects  should  be  developed  and  maintained,  for  better  management 
of  research  and  better  use  of  research  results. 

7.  AFOSR  should  ensure  that  a  good  quantitative  assessment  is  made 
of  future  Air  Force  science  and  engineering  work  force  needs.  This  assessment 
should  result  in  a  strategic  plan  for  the  support  and  development  of  faculty 
and  postdoctorate  and  graduate  students.  Until  the  plan  is  prepared,  AFOSR 
should  continue  its  current  levels  of  funding  for  university  faculty  and  science 
and  engineering  education  programs. 

8.  All  the  quality  metrics  identified  should  be  used  as  the  basis  of  an 
AFOSR  annual  report,  to  describe  the  quality  and  effectiveness  of  its 
program.  The  ongoing  application  of  such  metrics  will  allow  AFOSR  to  better 
manage  its  programs,  target  its  funds,  and  justify  the  value  of  its  work. 
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MEANS  AND  EXTENT  OF  TECHNOLOGY  TRANSITION 
BY  AFOSR  AND  PERFORMERS  OF  6.1  RESEARCH 

The  notional  "linear  model"  of  defense  R&D,  in  which  successful  6.1 
program  results  lead  to  exploratory  development  in  6.2  programs,  and  so  on, 
does  not  always  function  smoothly.  The  Air  Force  has  clearly  recognized  and 
addressed  this  problem.  Among  other  measures,  the  new  "relevancy"  ratings 
of  6.1  programs  by  the  Air  Force  laboratories  are  a  valuable  method  to  help 
ensure  the  ultimate  transitioning  of  6.1  results.  AFOSR  has  also  used  6.1 
dollars  on  occasion  to  encourage  transition,  as  for  a  recent  research 
consortium  on  microwave  tubes.  In  the  past  few  years,  AFOSR  has  also 
instituted  a  new  S&T  Coordinator  program,  in  which  selected  program 
managers  are  designated  to  facilitate  transition  between  discipline-oriented 
basic  research  and  the  multidisciplinary,  technology-oriented  efforts  of  the  Air 
Force  laboratories. 

Still,  these  valuable  steps  are  not  enough  to  solve  the  problem.  As 
discussed  immediately  above,  there  are  no  systematic  data  on  the  transition 
of  AFOSR-sponsored  research.  Thus,  no  real  evaluation  of  the  success  of  this 
transition  is  possible,  and  improving  transition  without  such  information  is  far 
more  difficult.  Again,  a  systematic  information  base  on  transition  could  be  of 
great  help  in  assessing  and  managing  the  process.  The  recent  establishment 
of  an  AFOSR  historian  should  be  one  useful  step  toward  developing  such  an 
information  base. 

Other  straightforward  methods  might  enhance  transition  and  could  be 
implemented  even  before  a  database  is  developed.  They  include  the  greater 
identification  of  potential  users  of  6.1  research  by  program  managers;  the 
regular  distribution  of  all  6.1  research  reports  to  all  identified  potential  users 
(such  distribution  is  currently  quite  variable);  encouraging  better 
communication  between  6.1  researchers  and  potential  users,  for  example, 
through  special  meetings  of  the  two  groups,  which  might  be  scheduled  at 
current  annual  review  meetings;  and  the  study  of  case  histories  in  transition, 
for  example,  in  the  recommended  ongoing  seminar  for  program  managers. 
Such  actions  could  build  on  the  S&T  Coordinator  program,  which  has  been 
active  in  a  number  of  these  areas. 


Recommendations 

1.  AFOSR  should  explore  the  development  of  a  systematic  information 
base  on  the  transition  of  its  basic  research  results.  The  Commander  of  AFMC 
should  develop  goals  and  guidelines  for  this  new  process  either  through  the 
AFMC  Technology  Executive  Officer  or  through  AFOSR. 
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2.  Additional  means  of  encouraging  technology  transition  should  be 
pursued:  distributing  6.1  research  reports  more  systematically,  encouraging 
further  meetings  between  researchers  and  potential  users,  and  examining  cases 
histories  in  transition  for  the  lessons  they  offer. 


EXTENT  OF  COVERAGE  BY  AFOSR  OF  TECHNICAL  AREAS 
THAT  HAVE  STRONG  INDUSTRY  RESEARCH  ACTIVITY 

The  committee  addressed  the  identification  of  research  redundancy  by 
considering  Air  Force  research  management  processes.  With  roughly  1,500  Air 
Force  basic  research  tasks  currently  underway,  identifying  research 
redundancy  is  a  daunting  task,  if  approached  from  the  top  down;  and  it 
certainly  carmot  be  done  by  one  small  group  of  experts. 

Of  greater  interest  to  the  Air  Force  is  how  to  avoid  unwarranted 
redundan^  of  its  basic  research  tasks  over  the  long  run,  rather  than  merely 
today.  This  goal  can  be  best  achieved,  again,  through  well-designed  review 
processes.  These  processes  should  ensure  that  the  views  of  appropriate  experts 
both  within  and  outside  the  Air  Force  are  obtained  about  the  value  of  specific 
research  tasks.  Experts  representing  the  specialized  fields  at  issue  can  provide 
the  best  advice  to  help  avoid  research  redundancy. 


Recommendation 

To  avoid  imwarranted  redundancy  of  Air  Force  and  industry  basic 
research,  the  Air  Force  should  strengthen  its  means  of  achieving  appropriate, 
ongoing  external  review  of  its  basic  research  projects  and  programs.^ 


TECHNICAL  AREAS  THAT  NEED  GREATER  COVERAGE 
AND  THOSE  RECEIVING  MORE  ATTENTION  THAN 
THE  RESEARCH  VALUE  WARRANTS 

The  committee  addressed  research  omissions,  like  research  redundancy, 
as  a  management  process  issue,  for  the  same  reasons.  A  number  of  the 
actions  recommended  throughout  the  report  could  help  identify  research 
omissions  as  well  as  research  redundancies  and  inappropriate  research 


^  The  Semiconductor  Industry  Association  (SIA)  is  attempting  to  establish  a  framework  to 
rationalLze  new  industry/govemment  technology  investments.  Examining  this  effort  was  beyond 
the  scope  of  the  present  study,  but  the  Air  Force  might  benefit  from  SIA’s  experience. 
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emphases.  Beyond  the  greater  involvement  of  external  peers  in  research 
review,  the  greater  involvement  of  the  laboratories  and  other  relevant  Air 
Force  groups  in  strategic  and  program  planning,  as  earlier  discussed,  would 
help  greatly  in  focusing  research  on  areas  of  greatest  need  and  opportunity  in 
the  context  of  the  Air  Force  mission. 


Recommendation 

To  ensure  the  best  basic  research  emphases,  the  Air  Force  should 
promote  the  greater  involvement  of  laboratories  and  other  appropriate  Air 
Force  groups  (c.g.,  operational  commands),  as  well  as  the  outside  research 
community,  in  strategic  and  program  planning  and  research  reviews. 


AFOSR  RESEARCH  MANAGEMENT  COMPARED 
TO  RESEARCH  MANAGEMENT  IN  OTHER  ORGANIZATIONS 

Research  practices  of  both  DOD  and  non-DOD  agencies  were  compared 
to  those  of  AFOSR.  Several  resulting  observations  have  already  been  made 
(see  discussion  of  the  &st  charge  above,  on  level,  management,  and  allocation 
procedures  for  6.1  resources). 

The  review  of  DOD  agencies  led  to  several  additional  observations. 
First,  it  is  difficult  to  achieve  highly  qualified,  stable  technical  leadership 
through  the  rotation  of  military  officers.  The  committee  also  heard  some 
reports  that  the  Air  Force  fails  to  look  on  research  management  assignments 
as  positive  military  career  moves  (though  this  view  was  not  universal).  At  the 
samP!  time,  AFOSR  reports  no  significant  difference  in  the  performance  of  its 
military  and  civilian  research  managers.  Some  experts  also  pointed  out  that 
military  research  managers  provide  a  better  perspective  on  users’  needs. 

The  second  issue  raised  in  the  review  of  other  DOD  agencies  was  the 
need  for  discretionary  laboratory  funds.  Numerous  published  reports  and 
comments  made  to  the  committee  suggest  that  discretionary  laboratory  funds, 
including  lUR  funds,  are  vital  to  the  laboratories’  flexibility  and  then- 
resulting  ability  to  pursue  innovative  work.  The  1983  Federal  Laboratory 
Review  Panel  suggested  that  5  to  10  percent  of  a  laboratory’s  funding  should 
be  discretionary,  a  value  range  with  which  this  committee  agrees.  At  the  same 
time,  visibility  on  such  discretionary  work  must  be  afforded  to  the  rest  of  the 
Air  Force  research  community,  in  particular  to  AFOSR,  to  allow  coordination 
and  avoid  redundancy. 

Third,  the  recently  implemented  Tri-Service  S&T  Reliance  program  was 
briefly  reviewed.  This  set  of  formal  agreements  and  processes  to  consolidate 
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and  collocate  selected  6.1,  6.2,  and  6.3A  S&T  programs  of  the  three  services 
was  implemented  in  late  1991,  under  the  direction  of  the  Joint  Directors  of 
Laboratories  (JDL).  Reliance  is  still  too  young  to  assess  fully,  but  it  clearly 
offers  valuable  opportunities  for  research  effectiveness  and  efficiency  across 
DOD  over  the  longer  term.  In  the  committee’s  view.  Reliance  as  a  whole  has 
gotten  off  to  a  variable  start;  initial  reports  do  indicate  that  it  is  providing 
identifiable  benefits  in  particular  areas  (see,  e.g..  Report  of  the  1992  Defense 
Science  Board  Task  Force  on  Microelectronics  Research  Facilities),  and  such 
successes  could  provide  the  basis  for  Reliance’s  future  development. 

Research  management  lessons  the  committee  gleaned  from  the  practices 
of  non-DOD  agencies  concern  mainly  two  areas;  the  use  of  external  review 
to  assure  program  quality,  and  the  necessity  of  ensuring  good  continuing 
technical  training  of  research  managers. 

Representatives  of  the  non-DOD  agencies  consulted  all  stressed  the 
value  of  their  external  review  processes.  All  these  agencies— DOE,  NASA,  and 
NSF— involve  external  reviewers  in  different  ways  in  their  proposal  and  project 
reviews.  All  report  that  these  reviews  significantly  enhance  both  the  quality 
and  credibility  of  their  programs.  These  various  review  processes  have 
different  strengths  and  weaknesses,  as  detailed  in  Appendix  B.  Comparative 
analysis  as  well  as  other  observations,  then,  suggest  that  the  Air  Force  might 
benefit  greatly  from  this  experience  and,  building  on  its  own  review  processes 
and  needs,  devise  better  external  review  processes  to  improve  its  research 
management. 

A  problem  commonly  reported  by  the  non-DOD  agencies  is  the  risk  of 
research  managers’  losing  their  technical  skills  because  of  administrative 
burdens.  This  was  not  reported  to  be  a  serious  problem  for  AFOSR  program 
meinagers,  but  it  should  be  carefully  avoided,  especially  insofar  as  Air  Force 
research  managers  have  very  significant  technical  decision-making  authority. 


Recommendations 

1.  The  Air  Force  should  continue  to  emphasize  recruitment  of  civilian 
research  managers  to  ensure  stable,  highly  qualified  technical  leadership.  For 
the  same  reason,  longer  tours  of  duty  should  be  considered  for  military 
research  managers.  Their  career  opportunities  should  be  ensured.  All  program 
managers  should  have  solid  training  and  experience,  including  technical 
management  training  and  3  to  5  years  of  technical  management  experience. 

2.  Laboratory  discretionary  funding,  including  ILIR  funding,  is  essential 
for  innovative  lab  work  and  should  be  protected.  At  the  same  time,  AFOSR 
should  be  given  review  responsibility  for  ILIR  work,  to  help  provide  a 
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common  standard  for  quality  and  to  assist  in  the  integrated  planning  of  Air 
Force  research,  especijJly  to  avoid  research  redundancy. 

3.  The  Air  Force  should  continue  to  pursue  opportunities  for  research 
efficiency,  as  through  strong  participation  in  such  cooperative  efforts  as  Tri- 
Service  S&T  Reliance.  Future  development  of  Reliance  should  build  on  its 
identified  successes,  and  its  effectiveness  should  continue  to  be  assessed  and 
ensured  through  appropriate  cross-service  studies. 

4.  Examination  of  the  practices  of  non-DOD  agencies  indicates  that  the 
Air  Force  should  incorporate  more  in-depth,  external  technical  review  in  its 
management  of  basic  research.  Improved  Air  Force  review  practices  should 
take  into  account  not  only  this  experience,  but  also  current  Air  Force 
practices  and  involved  groups.  Care  must  be  taken  to  ensure  that  external 
review  groups  have  a  clear  vision  of  potential  Air  Force  needs. 

5.  The  experience  of  other  federal  agencies  demonstrates  that  Air 
Force  progrjun  managers  must  be  given  the  workload,  span  of  control,  and 
professional  opportunities  to  allow  the  continual  renewal  of  their  technical 
skills. 


Appendix  A: 

Interactions  Between  the  Air  Force 
Office  of  Scientific  Research  and  the  Air  Force  Laboratories 

William  L.  Lehmann,  Chainnan 
Jack  L.  Kerrebrock 
George  H.  Miley 
Elizabeth  J.  Rock 

The  main  charge  to  the  subcommittee  was  to  assess  the  interactions  of 
the  Air  Force  Office  of  Scientific  Research  (AFOSR)  and  the  Air  Force 
laboratories  in  carrying  out  6.1  research  (specifically,  the  Defense  Research 
Sciences  program  element  6.1102F),  to  suggest  ways  of  improving  Air  Force 
basic  research  management.  The  subcommittee  was  also  mindful  of  the  study 
charges  to  assess  the  means  and  extent  of  technology  transition,  and  to 
consider  issues  of  research  funding,  redundancy,  and  leveraging. 

Policies  and  procedures  governing  the  interaction  of  Air  Force 
laboratories  with  AFOSR  were  covered  in  briefings  to  the  full  committee  by 
the  Director  of  Science  and  Technology  and  Chief  Scientist  of  Air  Force 
Systems  Command  (AFSC),  the  Director  of  AFOSR,  and  the  Chief  Scientist 
of  the  Wright  Laboratory,  the  Air  Force’s  largest  laboratory.  The 
subcommittee  also  made  site  visits  to  the  Rome,  Wright,  and  Phillips 
laboratories  and  attended  AFOSR  research  program  reviews  held  at  the 
Rome,  Phillips,  Wright,  and  Armstrong  laboratories.  In  the  course  of  these 
meetings  and  site  visits,  the  subcommittee  also  benefited  firom  discussions  with 
AFOSR  scientific  directors  and  program  managers  and  with  senior  laboratory 
scientists. 

The  environment  in  which  defense  research  is  conducted  and  managed 
has  changed  notably  in  recent  years.  There  have  been  downward  budgetary 
pressures  and  they  will  likely  continue.  The  threat  environment  has  changed 
dramatically.  Sophisticated  basic  research  conducted  outside  both  the  services 
and  the  nation  has  greatly  increased. 

Clearly,  past  Air  Force-sponsored  research  has  made  important 
contributions  to  the  service’s  present  military  capability.  Developments  in 
lasers  have  improved  precision-guided  munitions.  Advanced  weather  models 
have  led  to  better  target  acquisition  systems  and  combat  communications. 
Basic  research  in  aerodynamics  has  improved  the  performance  of  jet  engine 
compressors,  revolutionizing  fighter  airaaft  performance  and  range.  Life 
sciences  research  has  expanded  the  G-loading  envelope  in  which  pilots  can 
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operate.  Moreover,  in  the  past  25  years,  20  Air  Force-sponsored  U.S. 
researchers  have  won  Nobel  prizes. 

Even  in  the  face  of  such  achievements,  though,  it  is  valid  to  ask  whether 
current  research  management  policies  and  procedures  are  optimal,  especially 
given  the  new  defense  research  environment. 

In  any  restructuring,  several  overarching  principles  must  provide  the 
framework  for  AFOSR-laboratory  interactions.  Commitment  to  serving  the 
ultimate  customer,  the  operational  Air  Force,  is  essential,  especially  to 
facilitate  the  transition  of  basic  research  efforts  into  operational  capabilities. 
General  policy  and  doctrine  must  also  strongly  shape  research  management. 
For  example,  because  technological  superiority  is  critical  to  fulfillment  of 
national  policy,  the  Air  Force  must  be  an  active  member  of  the  basic  research 
community  spanning  Air  Force  laboratories,  the  defense  industry,  and 
universities.  Its  research  programs  must  serve  not  only  Air  Force  acquisition, 
but  also  Air  Force  logistics  and  operations. 

In  providing  specific  comments  below,  this  subcommittee  report  first 
outlines  the  organizational  structures,  roles,  policies,  and  procedures  that  are 
central  to  Air  Force  management  of  basic  research.  It  then  examines  the 
issues  raised  over  the  course  of  this  study  for  interactions  between  AFOSR 
and  the  laboratories,  and  offers  conclusions  about  means  of  improving  Air 
Force  research  management. 


OVERVIEW  OF  ORGANIZATIONAL  STRUCTURES,  ROLES, 
POLICIES,  AND  PROCEDURES 

A  successful  Air  Force  program  of  basic  research  requires  appropriate 
organizational  structures  and  well-defined  roles,  policies,  and  procedures. 


Structure  of  the  Defense  Research  and  Development  Program 

The  defense  research  and  development  program  is  based  on  a  "linear 
model":  Research  (program  6.1)  leads  to  Ejq)loratory  Development  (6.2), 
which  leads  to  Advanced  Development  (6.3),  then  in  turn  to  Engineering 
Development  (6.4),  and,  ultimately,  to  the  acquisition  of  new  weapon  systems, 
which  provide  the  basis  for  a  new  force  structure  (Figure  A-1).  This  "linear" 
R&D  model  is  notional,  of  course,  idealizing  the  true  R&D  process;  these 
stages  in  fact  interact  in  varied  ways,  with  feedback  loops  of  many  kinds. 

Other  defense  budget  programs  encompass  operations  and  support, 
including  logistics  and  basing.  Operations,  logistics,  and  basing  are  also  some 
of  the  defense  analogues  of  end  products  and  manufacturing  processes  in  the 
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FIGURE  A-1  The  Air  Force  research  and  development  process  (adapted  from  material  provided  by  Air  Force 
Systems  Command). 
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civil  sector.  Defense  research  is  expected  to  contribute  to  operations  and 
support,  as  well  as  to  the  acquisition  of  new  systems,  in  the  evolving  defense 
program  structure. 

Air  Force  basic  (6.1)  research  is  the  Defense  Research  Sciences  program 
element  (6.1102F)  ident^Ged  in  the  annual  DOD  budget.  By  guidance  from 
the  Director  of  Defense  Research  and  Engineering  (DDR&E),  it  is 
responsible  for  the  following: 

•  discovery  of  new  phenomena; 

•  visibility  and  exploitation  of  current  and  new  scientific  knowledge; 

•  filling  gaps  in  the  fundamental  technical  knowledge  base  in  areas 
important  to  Air  Force  operations  and  support; 

•  transition  of  research  results  to  further  Air  Force  operational 
capability;  and 

•  strengthening  the  research  infrastructure  in  Air  Force  laboratories, 
including  through  the  training  of  scientists  and  engineers  in  disciplines  critical 
to  Air  Force  needs. 


Air  Force  Command  Structure  for  Research  and  Development 

Air  Force  Materiel  Command  (AFMC)  is  the  Air  Force  organization 
responsible  for  the  development  and  acquisition  of  new  weapons  systems  and 
the  support,  maintenance,  and  modification  of  existing  weapon  systems.^ 
AFMC  elements  important  to  research  include  AFOSR,  which  manages  all 
Air  Force  basic  research,  both  intra-  and  extramural,  and  the  laboratories  of 
the  four  Air  Force  product  centers— Wright  Laboratory  of  the  Aeronautical 
Systems  Center,  Rome  Laboratory  of  the  Electronic  Systems  Center,  Phillips 
laboratory  of  the  Space  and  Missile  Systems  Center,  and  Armstrong 
Laboratory  of  the  Human  Systems  Center— which  manage  or  conduct  research 
at  all  stages  of  R&D  through  advanced  development. 

AFMC  also  has  five  Air  Logistics  Centers,  with  their  engineering 
organizations,  which  apply  existing  and  new  technology  for  maintenance  and 
modernization.  New  ideas  and  technology  initiatives  ft^equently  come  from 
these  centers,  are  encouraged,  and  are  exceptions  to  the  "linear  model."  There 
are  currently  no  formal  ties  between  AFOSR  and  the  Air  Logistics  Centers. 

The  AFMC  Commander  relies  on  a  Director  of  Science  and  Technology 
and  Director  of  Engineering  for  matters  pertaining  to  research.  The  first  is 


^  Over  the  course  of  this  study,  the  original  sponsor,  Air  Force  Systems  Command  (AFSC) 
was  merged  with  Air  Force  Logistics  Command  (AFLC)  to  form  the  new  Air  Force  Materiel 
Command.  The  subcommittee  has  adapted  its  commentary  accordingly. 
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responsible  for  the  technoloar  base  (programs  6.1,  6.2,  and  part  of  6.3).  The 
second  is  responsible  for  engineering  and  support,  and  is  a  primary  customer 
of  the  technology  base. 


Program  Acquisition  Structure 

The  acquisition  of  new  weapon  systems  is  managed  by  Acquisition 
Executives  in  each  service,  in  the  case  of  the  Air  Force,  the  Assistant 
Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ).  SAF/AQ  is  responsible 
for  all  Air  Force  6.1,  6.2,  6.3,  and  6.4  program  elements. 

SAF/AQ  has  designated  the  AFMC  Director  of  Science  and  Technology 
as  his  Technolo©'  Executive  Officer  (TEO),  with  program  responsibility  for 
Air  Force  6.1,  6.2,  and  certain  6.3  programs.  The  TEO  briefs  SAF/AQ  on  6.1 
research,  which  is  represented  by  means  of  the  Research  Technology  Area 
Plan  (TAP),  one  of  a  dozen  such  plans. 


Air  Force  Basic  Research  Program  and  Organization 

AFOSR  has  "single  manager  responsibility"  for  the  6.1  Defense  Research 
Sciences  program  (funded  at  about  $203  million  in  fiscal  year  1992)  and  the 
6.1  University  Research  Initiative  program  (funded  at  about  $22  million), 
reporting  to  the  Commander  AFMC  through  the  Director  of  Science  and 
Technology.  The  6.1  In-House  Laboratory  Independent  Research  (ILIR) 
program  ($8  million)  is  the  responsibility  of  the  lab  directors. 

The  Air  Force  Defense  Research  Sciences  program  is  broken  down  into 
15  projects.  Each  project  (funded  at  $5-40  million)  is  associated  with  a  single 
research  area,  such  as  structural  materials,  atmospheric  sciences,  mathematical 
and  computer  sciences,  or  physics. 

Each  project  area  in  turn  is  broken  into  individual  grants,  contracts,  and 
laboratory  tas^.  There  are  some  1,400  grants  and  contracts  (averaging  about 
$100,000)  with  universities  and  industry,  and  some  100  intramural  research 
tasks  with  Air  Force  laboratories  (averaging  $400,000).  AFOSR  guidance 
indicates  that  intramural  investment  should  represent  about  one-third  of  the 
Air  Force’s  Defense  Research  Sciences  funding  (see  Attachment  A-1,  which 
provides  an  overview  of  roles  and  responsibilities  relating  to  Air  Force 
intramural  basic  research). 

AFOSR’s  "single  manager  responsibility"  encompasses  the  formulation 
of  the  Air  Force  Research  program,  coordination  of  intra-  and  extramural 
research,  resource  allocation,  progress  monitoring,  review  and  evaluation  of 
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results,  and  transition  to  Air  Force  benefit.  AFOSR  also  selects  proposals  and 
awards  research  grants  and  contracts  to  universities  and  industry. 

The  four  Air  Force  laboratories  are  charged  with  building  and 
maintaining  the  technology  base  appropriate  to  their  parent  centers:  each 
laboratory  conducts  in-house  work  in  Research  (6.1),  on  the  order  of  $10 
million;  carries  out  in-house  and  contractual  efforts  in  Exploratory 
Development  (6.2),  on  the  order  of  $100  million;  and  formulates  and  manages 
contractual  efforts  in  Advanced  Development  (6.3),  on  the  order  of  $100 
millioiL  Individual  laboratory  manning  is  on  the  order  of  1,500  to  2,500,  plus 
on-site  support  contractors. 

Each  laboratory  has  a  commander  or  director,  assisted  by  a  laboratory 
chief  scientist,  and  from  5  to  10  directorates,  each  with  its  own  associate  chief 
scientist  (associate  chief  scientists  report  to  the  chief  scientists  of  their 
respective  laboratories). 

To  formulate  the  Air  Force  Research  program  for  the  fiscal  year,  the 
AFOSR  Director  receives  armual  policy  guidance  and  resource  direction  from 
the  AFMC  Director  of  Science  and  Technology,  and  is  assisted  in  program 
planning  by  the  five  AFOSR  scientific  directors,  a  Research  Council  consisting 
of  himself  and  the  chief  scientists  of  the  four  laboratories,  and  his  staff. 
Together  they  monitor  progress  and  review  and  evaluate  results  of  the 
research  projects,  and  report  annually. 

Each  of  the  five  AFOSR  scientific  directors  has  responsibility  for  one  or 
more  of  the  research  projects.  The  scientific  director  is  assisted  in  this  by 
appropriate  lab  directorate  chief  scientists  and  5  to  10  AFOSR  program 
managers  (there  are  a  total  of  40  some  program  managers  overall).  Each 
program  manager  selects  fi’om  20  to  40  proposals  for  contract  or  grant  award. 

In  addition,  the  Air  Logistics  Center  engineering  groups,  which  assure 
the  maintenance  and  quality  of  existing  aircraft  and  missiles,  also  benefit  from 
technical  partnerships  with  Air  Force  laboratories  and  AFOSR. 

AFOSR  also  has  a  small  in-house  research  program  at  the  Seiler 
Laboratory  at  the  Air  Force  Academy  and  research  liaison  offices  in  Europe 
and  Japan. 

Additional  roles,  policies,  and  procedures  relating  to  Air  Force 
management  of  basic  research  are  described  under  the  discussion  of  issues 
below. 


ISSUES 

Based  on  briefings  and  site  visits,  the  subcommittee  identified  a  number 
of  important  issue  areas.  Each  is  discussed  below,  and  conclusions  are  offered 
for  the  questions  that  arose  concerning  each  area. 
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Air  Force  Funding  of  6.1  Research 

•  Is  the  current  Air  Force  RDT&E  investment  in  6.1  research  appropriate? 

•  Is  the  present  division  of  6.1  funding  between  intra-  and  extramural 
research  appropriate? 

The  fraction  of  Air  Force  RDT&E  funding  that  goes  to  6.1  research  is 
only  about  half  of  the  corresponding  value  for  Army  and  Navy  6.1  research 
(see  Appendix  B,  Figure  B-5).  No  clear  explanation  for  this  discrepancy  was 
presented  to  the  subcommittee.  Thus,  whether  the  Air  Force  is  investing 
appropriate  resources  in  6.1  research  is  an  open  question. 

AFOSR  1992  "Guiding  Principles"  for  the  intramural  basic  research 
program  (Attachment  A-1)  suggest  that  about  one-third  of  basic  research 
funding  should  go  to  the  intramural  program,  but  that  specific  distributions 
may  deviate  substantially,  depending  on  technical  and  management  factors, 
to  optimize  overall  benefits. 

More  basic  research  funding  is  designated  to  extramural  projects,  to  tap 
the  expertise  of  universities  and  industry.  At  the  same  time.  Air  Force 
laboratory  staff  report  that,  to  maintain  strong  in-house  research  teams,  they 
need  more  funding.  Intramural  research  is  in  fact  conducted  in  large  part  to 
enable  the  laboratories  to  absorb  outside  advances  effectively.  Changes  in  the 
division  of  funds  between  internal  and  external  research  would  thus  have  both 
advantages  and  disadvantages.  However,  no  persuasive  arguments  were 
presented  to  the  committee  that  a  significant  change  in  this  division  is  needed. 


Conclusions 

In  view  of  the  practices  reported  by  the  other  services,  the  subcommittee 
feels  that  the  Air  Force  administration  should  consider  increasing  the 
percentage  of  RDT&E  funds  that  go  to  6.1  research.  Indeed,  if  this  is 
possible,  some  of  the  problems  reported  by  the  laboratory  staff  can  be 
alleviated  without  changing  the  balance  of  intra-  and  extramural  funding. 


Strategic  Planning 

•  Is  the  general  approach  to  6.1  strategic  planning  optimal? 

•  Are  Air  Force  senior  leaders  sufficiently  cognizant  of  Air  Force  basic 
research  programs? 
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A  number  of  strategic  planning  processes  shape  the  development  of  Air 
Force  basic  research  programs:  formal  guidance  received  from  SAF/AQ  and 
DDR&E,  preparation  of  the  Air  Force  Research  TAP,  Tri-Service  S&T 
Reliance  documents  and  the  five  complementary  cross-cutting  interlaboratory 
investment  plans  (ILIPs,  in  the  areas  of  electron  devices,  software, 
producibility,  transient  electromagnetics,  and  environment,  safety,  and 
occupational  health),  and  AFOSR’s  Spring  Review  presentation  to  HQ 
AFMC/S&T  Corporate  Board.  DOD  has  identified  a  set  of  critical 
technologies,  common  to  all  services,  that  are  discipline-  or  topic-  rather  than 
mission-oriented;  each  service  must  relate  its  research  program  to  these  DOD 
critical  technologies. 

However,  for  all  these  processes’  value,  they  do  not  provide  adequate 
mission-specific  strategic  planning  for  the  Air  Force.  The  unique  advantage 
of  Air  Force  research  and  research  management  is  the  ability  to  conduct  and 
capitalize  on  the  research  and  technology  that  is  critical  to  Air  Force  needs. 

For  example,  given  the  Air  Force  planning  blueprint  of  "Global 
Reach— Global  Power"  and  present  programmed  resources,  the  Air  Force  will 
be  flying  many  of  its  current  aircraft  well  into  the  21st  century.^  Achieving 
this  goal  will  require  strong  maintenance  and  technology  insertion  programs 
to  ensme  viability,  survivability,  and  effectiveness.  The  Air  Force  now  has 
such  programs,  in  maintenance  in  the  operational  commands,  in  modification, 
repair,  and  remanufacturing  in  the  Air  Logistics  Centers,  in  engineering  in  the 
product  centers,  and  in  technology  in  the  laboratories.  Its  long-established 
Aircraft  Structural  Integrity  Program  (ASIP)  is  a  recognized  model  for  the 
technical  management  of  aging  aircraft,  not  only  by  the  Air  Force,  but  also  by 
DOD,  FAA  NASA,  and  others. 

Yet  there  are  still  important  technical  unknowns  in  supporting  aging 
aircraft  (e.g.,  the  relation  of  micro-  to  macroscopic  material  properties  in  the 
formation  and  propagation  of  cracks,  the  behavior  of  structures  composed  of 
different  materials  in  which  there  is  multiple-site  damage,  and  some  of  the 
fundamental  phenomena  of  interest  for  corrosion  and  nondestructive 
evaluation).^  Many  of  these  technical  unknowns  of  particular  interest  to  the 
Air  Force  must  be  addressed  through  basic  research.  The  Air  Force  has 
already  made  commendable  efforts  in  its  emphasis  on  some  of  these  areas  of 
research. 


^  See  Global  Reach-Global  Power  The  Evolving  Air  Force  Contribution  to  Natiorml  Security, 
White  Paper,  Department  of  the  Air  Force,  December  1992. 

^  While  AFOSR  supports  little  research  in  the  area  of  nondestructive  evaluation,  it  provided 
important  support  to  a  successful  recent  workshop  sponsored  at  an  Air  Logistics  Center,  with 
participation  of  the  FAA,  NASA,  DOD,  and  others. 
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A  "Strategy  to  Task"  approach,  in  which  operators  and  researchers 
exchange  their  experience  and  knowledge,  has  been  used  with  some  success 
for  mission-oriented  technology  planning.^  This  approach  might  be  usefully 
applied  to  planning  Air  Force  basic  research. 

Strategic  research  planning  in  the  new  environment  will  require  an 
increased  focus  on  Air  Force  mission  needs,  but  it  should  be  emphasized  that 
this  focus  should  not  lead  to  using  6.1  funds  for  applied  research.  Basic 
research  can  be  more  oriented  to  mission  needs  and  still  be  fundamental 
science  (e.g.,  basic  science  research  relating  to  corrosion).  Moreover,  to 
achieve  the  revolutionary  advances  of  the  future,  some  basic  research  must  be 
strictly  science-driven,  pursued  without  concern  for  application. 


Conclusions 

Especially  in  view  of  constrained  defense  budgets,  strategic  planning  for 
basic  research  should  take  better  account  explicitly  of  long-term  Air  Force 
mission  goals.  Good  planning  now  requires  strong  mission  as  well  as  discipline 
orientations.  Mission-oriented  planning  should  reflect  the  judgment  of 
AFOSR,  the  laboratories,  the  Air  Logistics  Centers,  the  operational  Air  Force 
commands,  and  other  representatives  of  the  Air  Force  R&D  community.  Such 
strategic  planning  would  help  identify  and  pursue  the  technical  thrusts  and 
research  programs  that  may  best  benefit  the  future  Air  Force  (e.g.,  a  focused 
research  program  on  the  problems  of  aging  aircraft). 

A  small  portion  of  Air  Force  basic  research  funding  should  be  set  aside 
to  pursue  especially  innovative  research,  regardless  of  any  applications  it 
promises. 

The  practice  of  briefing  the  basic  research  strategic  plan  to  senior  Air 
Force  leaders  at  AFMC  and  Air  Force  Headquarters  is  strongly  endorsed, 
along  with  other  measures  that  enhance  high-level  understanding  of  the 
potential  of  basic  research. 


Development  of  Research  Programs 

•  How  can  the  formulation  of  Air  Force  basic  research  progratm  be 
improved? 

•  Should  the  Air  Force  laboratories  play  a  greater  role  in  program 
development? 


^  See  Glenn  A.  Kent  and  William  E.  Simons,  1991,  A  Framework  for  Enhandng 
Operational  Capabilities,  RAND,  Santa  Monica,  CaUfomia. 
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•  Should  there  be  greater  external  peer  input  to  program  development? 

•  How  can  AFOSR  best  exploit  new  areas  of  research,  if  these  areas  have 
no  AFOSR  advocate? 

Fonnulation  of  Air  Force  basic  research  programs  is  now  primarily  a 
top-down  process.  While  they  receive  guidance  from  a  number  of  sources, 
AFOSR  management  staff  largely  formulate  the  research  program,  which  is 
then  conveyed  to  the  Air  Force  laboratories  and  extramural  researchers. 

According  to  AFOSR’s  Guiding  Principles  (Attachment  A-1),  AFOSR 
scientific  directors  make  the  decisions  on  funding  shifts,  policy  matters,  and 
program  quality.  The  AFOSR  program  manager  is  responsible  for  funding 
decisions,  annual  reporting,  and  initiation  and  guidance  of  research,  and 
serves  as  the  principal  AFOSR  link  to  the  Air  Force  laboratories. 

Two  kinds  of  advisory  bodies  have  been  formed  to  assist  AFOSR  in 
program  planning,  the  AFOSR  Research  Council,  which  advises  the  AFOSR 
Director,  and  the  Technical  Advisory  Boards,  which  advise  AFOSR  scientific 
directors.  However,  the  program  manager  still  appears  to  be  the  primary 
interface  with  the  labs. 

Once  general  research  plans  are  formulated,  the  Air  Force  laboratories 
(and  separately,  external  labs)  propose  specific  research  within  the  identified 
interest  areas.  Again,  AFOSR  makes  the  final  funding  decisions.  These 
decisions  are  sometimes  made  in  consultation  with  the  laboratories,  but  in 
other  cases  they  are  made  effectively  by  AFOSR  program  managers  alone. 
Discussions  with  Air  Force  laboratory  staff  indicated  they  strongly  desire  to 
make  greater  contributions  to  this  process. 

Based  on  all  the  information  provided,  the  subcommittee  believes  that 
there  could  be  greater  partnership  between  AFOSR  and  the  laboratories  in 
formulating  research  programs.  There  is  some  danger  that  AFOSR  may  be 
acting  at  times  in  an  isolated  or  autocratic  maimer,  and  Air  Force  laboratory 
persoimel  could  contribute  greater  knowledge  of  special  Air  Force  technical 
requirements,  beneficially  shaping  basic  research  programs.  AFOSR  should 
fully  exploit,  insofar  as  possible,  the  specialized  expertise  it  has  at  hand. 
Additionally,  laboratory  staff  would  likely  be  better  facilitators  of  the 
transition  of  research  results,  when  they  have  had  a  greater  hand  in  planning 
the  research. 

At  the  same  time,  there  are  potential  dangers  in  seeking  greater 
laboratory  input:  potential  conflicts  of  interest  (which  may  be  worsened  by  the 
budgetary  pressures  that  lead  to  emphasizing  in-house  research  at  the  expense 
of  extramimal  research),  a  slide  into  outmoded  research  interests  or 
approaches,  and  the  undesirable  diffusion  of  leadership  (e.g.,  in  the  form  of 
poor  coordination  of  intra-  and  extramural  work,  or  of  tolerance  of  weak 
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components).  In  short,  greater  input  from  laboratory  persoimel  would  be 
valuable,  but  should  be  sought  with  care. 

AFOSR  research  managers  also  vary  greatly  in  the  way  they  seek 
external  input  in  program  formulation.  (As  the  Guiding  Principles  point  out, 
"excellence  is  determined  by  the  AFOSR  scientific  directors,  augmented  by 
peer  reviews  as  necessary.")  In  some  AFOSR  areas,  such  as  Life  and 
Environmental  Sciences,  Chemical  Sciences,  and  Atmospheric  Sciences, 
external  peer  review  of  proposals  is  a  formalized  process.  AFOSR  personnel 
involved  in  these  more  formal  reviews  observe  that,  for  their  own  areas  of 
research,  these  reviews  are  quite  helpful  in  providing  independent,  unbiased 
input,  a  view  in  keeping  with  the  consensus  and  practice  of  the  larger  research 
commimity. 

In  AFOSR  research  management  more  generally,  however,  external  peer 
review  is  obtained  only  ad  hoc.  This  fact  is  of  particular  concern  to  the 
subcommittee.  Extern^  input  to  Air  Force  research  programs  can  be 
invaluable  to  ensure  that  programs  remain  at  the  leading  edge  of  research, 
leverage  the  work  of  others,  avoid  redundancy,  and  are  of  the  highest  quality 
in  other  respects.  As  noted  by  AFOSR  personnel  and  many  others,  formalized 
external  reviews  can  present  their  own  problems,  including  lack  of  flexibility, 
burdens  of  time,  expense,  and  administrative  work,  and  the  potential 
introduction  of  biases,  as  in  any  review. 

These  considerations  together  lead  the  members  of  this  subcommittee 
to  hold  a  range  of  views.  At  one  end  of  the  spectrum,  some  think  that  external 
peer  reviews  should  be  regularized  across  AFOSR.  At  the  other,  some  think 
that  formal  arrangements  might  inhibit  the  creativity  and  motivation  of 
AFOSR  research  managers,  and  that  therefore  a  better  approach  would  be 
to  encourage  program  managers  to  achieve  optimal  external  input,  in  their 
own  chosen  ways,  through  continuing  education,  performance  reviews,  and 
salary  determinations.  The  use  of  research  databases,  especially  those  of  the 
other  services  and  relevant  federal  agencies,  could  provide  another  vtiluable 
type  of  information  about  external  research.  Regardless  of  specific  view, 
however,  all  subcommittee  members  agree  on  the  value  of  well-designed 
external  input  to  AFOSR  program  planning,  for  all  the  reasons  noted  above. 
The  group  as  a  whole  also  found  that  such  input  appeared  weak  in  some 
AFOSR  areas. 

Given  the  very  significant  technical  decision-making  authority  of  AFOSR 
research  managers— which  would  remain  even  with  the  changes  suggested 
here— great  effort  should  go  toward  ensuring  that  the  best  people  are  hired  for 
these  positions  and  that  they  have  the  resources  and  performance 
requirements  to  do  an  optimal  job. 

Again,  because  of  centralization  of  authority  in  Air  Force  basic  research 
planning,  there  is  greater  risk  that  leading-edge  and  other  important  research 
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may  go  unfunded.  Basic  research  is  increasingly  widely  conducted  and  diverse. 
Yet  if  proposed  research  is  not  of  direct  interest  to  an  AFOSR  program 
manager,  support  for  the  research  is  likely  to  be  minimal.  AFOSR  has  been 
very  slow  to  move  into  new  areas  of  research,  even  when  given  guidance  to 
do  so.  SAF/AQ  provided  guidance  to  build  research  supporting  logistics 
several  years  ago.  But  progress  has  been  delayed  because  no  program 
manager  understands  or  advocates  such  research. 

Additionally,  while  well-defined  steps  have  been  taken  to  encourage 
multidisciplinary  research,  current  basic  research  in  areas  bearing  on  logistics 
and  manufacturing  may  not  be  sufficient  (again,  the  Air  Force  would  benefit 
from  strong  programs  in  such  areas  as  corrosion,  stress-cracking,  tribology, 
artificial  intelligence  for  monitoring  systems,  and  nondestructive  evaluation). 
Greater  attention  is  being  given  to  such  research  by  DARPA  and  universities, 
and  the  Air  Force  should  be  concerned  to  pursue  and  capitalize  on  this  work. 

In  short,  the  present  means  of  formulating  the  Air  Force  basic  research 
program  has  the  advantages  of  strong  centralized  leadership,  but  the 
disadvantages  of  necessarily  quite  limited  scope.  It  would  appear  that  some 
changes  in  the  process  might  reduce  the  problems  noted. 


Conclusions 

The  Air  Force  should  devise  procedures  to  involve  at  the  "partnership 
level"  parties  beyond  its  own  staff  in  research  program  formulation. 
Laboratory  personnel  and  representatives  of  both  universities  and  industry 
where  relevant  work  is  funded  should  be  part  of  the  partnership.  In  particular, 
external  review  of  proposals  should  be  encouraged  through  appropriate 
means.  (Other  specific  possibilities  might  include,  as  one  laboratory  scientist 
suggested,  the  naming  of  lab  "adjunct  program  managers"  to  work  in 
partnership  with  AFOSR  staff.)  In  any  new  arrangements,  however,  care 
should  be  taken  to  avoid  the  pitfalls  noted  above  and  to  maintain  the  valuable 
coordination  that  AFOSR  provides. 

The  incorporation  of  critical  new  research  areas  should  also  be  kept  in 
mind  in  revising  program  formulation  procedures.  For  example,  an 
interdisciplinary  team  of  program  managers  could  be  designated  to  evaluate 
proposals  in  emerging  areas  of  potential  importance,  with  the  help  of 
appropriate  outside  expertise,  from  the  Air  Force  laboratories  and  the 
external  research  community.  A  small  percentage  of  the  6.1  budget  might  also 
be  held  back  each  year  to  allow  the  pursuit  of  new  areas.  Decisions  about 
such  areas  could  be  made  by  the  interdisciplinary  team. 
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Balance  of  Nearer-  and  Longer-Term  Research  Goals 

•  Does  the  Air  Force  6.1  program  successfully  balance  the  pursuit  of  both 
nearer-  and  longer-term  goals? 

Balancing  shorter-  and  longer-term  goals  is  a  traditional  problem  faced 
by  research  managers.  The  distinction  between  basic  work  and  technologr,  on 
the  one  hand,  and  development,  on  the  other,  is  not  always  clear.  For  that 
matter,  there  is  no  clear  consensus  among  managers  or  scientists  about  the 
definition  of  "basic  research."  Still,  some  reasonable  guidelines  can  be 
developed  to  help  avoid  confusion  concerning  these  terms. 

Regardless  of  definitional  uncertainties,  there  is  no  evidence  that  the  Air 
Force  has  significantly  shifted  away  fi'om  a  strong  basic  research  program. 
However,  two  related  questions  arose  in  the  subcommittee’s  fact-finding  about 
the  current  program.  First,  some  of  those  consulted  expressed  concern  that 
some  AFOSR  program  managers  improperly  interpret  basic  engineering 
science  research  as  "too  applied"  to  be  considered  basic  research.  There  is 
evidence  to  believe,  for  example,  that  a  number  of  good  extramural  research 
projects  have  been  rejected  via  such  reasoning.  Others  expressed  concern  that 
pressures  within  the  Air  Force  lead  some  in-house  labs  to  overemphasize 
near-term  results,  forcing  a  selection  of  projects  that  might  properly  be  called 
"applied." 


Conclusions 

AFOSR  administration  should  develop  its  policies  and  procedures  to 
help  program  managers  avoid  arbitrary  decisions  in  determining  the  "basic" 
or  "developmental"  nature  of  proposed  research.  Improving  this  decision 
making  will  likely  require  ongoing,  open  discussion  among  AFOSR  staff, 
inasmuch  as  one  natural  inclination  is  to  regard  the  work  one  knows  well  as 
more  ’basic,"  and  other  work  as  more  "applied."  One  good  management 
response  might  be  to  develop  a  seminar  series  for  AFOSR  research  managers. 
Program  managers  could  confront  this  and  a  number  of  other  important  issues 
as  part  of  such  a  "continuing  education"  program. 

One  particular  Air  Force  procedure  of  interest  is  the  annual  relevancy 
review.  Relevancy  to  the  Air  Force  mission  is  certainly  an  important  factor  in 
judging  the  value  of  AFOSR-sponsored  work.  However,  "relevancy"  should 
not  be  confused  with  "promising  shorter-term  results";  that  is,  a  higher  weight 
should  not  automatically  be  given  to  research  that  should  have  a  more 
immediate  payoff.  The  fundamental  research  underpinning  broad  areas  of 
science  and  technology— which  ultimately  supports  new  and  revolutionary 
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technologies— is  as  relevant  as  fundamental  research  in  support  of  current 
problem-solving. 


Technology  Transition 

•  How  can  Air  Force  transition  of  basic  research  results  to  the  Air  Force 
technology  base  be  improved? 

The  linear  model,  in  which  useful  6.1  research  results  are  presumably 
quickly  recognized  by  6.2  workers,  and  so  on  throughout  development,  does 
not  always  work  smoothly.  Subcommittee  members’  site  visits  (e.g.,  to 
Hanscom  Air  Force  Base)  suggested  that,  in  some  cases,  6.1  researchers  work 
closely  and  effectively  with  developers.  However,  sometimes  laboratory 
researchers  are  not  in  touch  with  6.2  people  in  the  same  lab.  The  6.2  and  6.3 
engineers  frequently  have  closer  ties  to  engineers  in  industry  or  universities. 
(In  fact,  the  nature  of  different  projects  alone  tends  to  produce  some  variation 
in  the  communicatio  of  6.1  researchers  and  other  Air  Force  researchers.) 

Air  Force  management  recognizes  this  problem.  Thus,  on  occasion, 
AFOSR  has  used  6.1  dollars  to  help  the  transition,  as  for  a  recent  research 
consortium  of  university  and  industry  groups  addressing  microwave  tubes, 
funded  by  AFOSR,  and  managed  by  Rome  Laboratory.  Additionally,  in  the 
past  two  years  AFOSR  has  established  an  S&T  Coordinator  program,  in 
which  selected  program  managers  are  designated  to  facilitate  technology 
transition  between  discipline-oriented  AFOSR  basic  research  and  the 
multidisciplinary,  technology-oriented  efforts  of  the  Air  Force  laboratories.^ 

A  difficulty  encoimtered  by  the  subcommittee  in  their  review  of 
transition  is  that  there  is  no  database  for  evaluating  how  effective  the 
transition  process  has  been.  Examples  of  successful  transitions  can  be  cited, 
but  such  examples  do  not  provide  an  adequate  overview  or  measure  of  the 
success  of  the  Air  Force  transition  process. 


Conclusions 

AFOSR  should  continue  to  focus  on  the  issue  of  transition.  One  way  to 
do  this  would  be  to  build  on  the  current  S&T  coordinator  program  (the 
experience  and  achievements  of  this  group  should  in  any  event  be  considered 


^  See  also  Helmut  Hellwig,  Transitioning  Research  Results:  A  Challenge  in 
Conmnmication,"  Proceedings  of  the  International  Conference  on  Management  of  Engineering  and 
Technology,  October  27-31, 1991. 
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in  the  future  development  of  Air  Force  transition  efforts).  The  development 
of  a  quantitative  database  on  past  and  continuing  progress  in  technology 
transition  is  one  important  question  to  explore.  K  a  measure  of  the 
effectiveness  of  existing  transition  mechanisms  can  be  established,  more 
logical  efforts  can  be  applied  to  improving  them. 

The  Commander  of  AFMC  should  develop  goals  and  guidelines  for  the 
new  process  of  gathering  information  on  transition,  either  through  the  AFMC 
Technology  Dcecutive  Officer  or  through  AFOSR.  The  recent  establishment 
of  an  AFOSR  historian  should  also  be  a  valuable  contribution  toward 
developing  this  information  base. 

Some  fairly  straightforward  methods  might  enhance  transition  and  could 
be  implemented  even  before  any  database  is  developed.  One  is  to  ensure  that 
all  program  managers  have  up-to-date  lists  of  potential  users  of  6.1  research 
within  their  areas.  Appropriate  research  reports,  for  both  in-house  and 
external  projects,  should  be  distributed  to  the  lists.  Such  report  distribution  is 
currently  quite  variable,  depending  on  the  practice  of  the  individual  program 
manager. 

Another  opportunity  to  encourage  communication  between  6.1 
researchers  and  potential  users  occurs  during  the  annual  review  meetings.  For 
example,  spring  and  fall  reviews  could  be  planned  so  that  potential  users 
could  participate  in  discussions  of  6.1  research  projects.  Such  meetings  could 
be  widely  publicized  and  held  at  convenient  locations,  with  sufficient  time 
scheduled  for  both  planned  and  informal  discussions  among  basic  researchers 
and  users. 

All  AFOSR  program  managers  should  be  encouraged  to  help  establish 
communication  between  researchers  and  potential  users.  Often  researchers  do 
not  know  who  possible  users  are;  the  AFOSR  manager  must  take  an  active 
part  in  helping  work  out  such  contacts.  Air  Force  laboratory  staff  are  clearly 
one  important  resource  that  the  AFOSR  manager  could  use. 

The  issue  of  transition  would  be  an  important  topic  in  the  "continuing 
education"  program  for  AFOSR  managers  proposed  earlier.  It  would  be 
instructive  to  present  case  histories,  so  that  program  managers  could  learn 
more  about  the  techniques  that  have  been  most  successful  and  about 
problems  that  have  hindered  transition. 


Best  Use  of  Diverse  Research  Groups 

•  How  can  the  Air  Force  best  use  the  spectrum  of  research  groups,  within 
in-house  laboratories,  universities,  and  industry  laboratories,  that  it  has  at  its 
disposal? 
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At  present.  Air  Force  research  projects  could  be  ranged  along  a 
continuum,  with  one  extreme  represented  by  the  work  of  individual 
researchers  for  small  groups,  and  the  other  by  collaborative  teams  involving 
Air  Force  laboratory,  industry,  and  university  researchers.^  Again,  the 
optimization  of  such  types  of  groups  is  a  classic  problem  in  research 
management 

The  subcommittee  did  not  find  any  evidence  that  the  current  Air  Force 
mix  is  inappropriate.  However,  some  concern  was  raised  by  the  fact  that  the 
mix  differs  considerably  among  program  managers,  depending  on  the 
manager’s  style  and  personal  views  about  research.  This  variation  is  not  bad 
in  itself,  but  if  managers  can  comfortably  handle  only  individual  investigators 
or  large  research  groups,  this  can  be  restrictive  and  fail  to  capitalize  on  the 
best  research  expertise. 


Conclusions 

AFOSR  should  ensure  that  program  managers  understand  how  to  handle 
both  small  research  groups  and  integrated  research  teams.  Information  on 
related  management  techniques  (e.g.,  case  studies  of  successful  integrated 
teams)  could  be  incorporated  in  the  "continuing  education  program"  for 
program  managers. 

At  the  same  time,  other  approaches  should  be  pursued  to  encourage 
continual  exchange  of  information  among  Air  Force  laboratories,  universities, 
and  industry,  so  teams  can  develop  through  a  self-motivation  process. 
Examples  of  such  approaches  are  the  AFOSR-sponsored  Summer  Faculty 
Research  Program  (SFRP)  and  the  University  Resident  Research  Program 
(URRP).  AFOSR  should  study  these  programs  to  ensure  they  are  successful 
in  building  good  working  relationships  between  university  and  Air  Force  staff. 
For  example,  one  goal  for  the  siunmer  faculty  program  should  be  that 
university  participants  continue  to  interact  with  Air  Force  laboratory 
personnel  after  returning  to  universities  (e.g.,  through  the  participation  of 
university  participants  in  the  Air  Force  funded  research  program).  How  well 
these  exchange  programs  are  working  in  building  such  relationships  is  unclear, 
because  no  appropriate  information  base  is  available  to  make  the  judgment. 
The  subcommittee  suggests  that  AFOSR  develop  this  kind  of  database,  so 
that  such  programs  can  be  evaluated. 

The  subcommittee  noted  that  there  is  no  counterpart  program  to  bring 
industry  research  staff  into  Air  Force  laboratories.  In  general,  owing  to  the 


®  Interservice  cooperative  research  is  discussed  in  Appendix  B. 
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extensive  interactions  of  Air  Force  researchers  with  industry  at  the  6.2  and  6.3 
levels,  it  is  easier  to  open  communication  channels  with  industry  than  with 
universities  (which  typically  do  not  participate  in  these  later  development 
programs).  Still,  AFOSR  should  consider  whether  some  additional  mechanism 
or  program  should  be  used  to  encourage  Air  Force  laboratory-industry 
interactions  at  the  basic  research  level. 


Quality  Laboratory  Researchers 

•  What  approaches  can  be  used  to  attract  and  retain  high-quality 
researchers  in  the  Air  Force  laboratories? 

The  vitality  of  the  Air  Force  Research  program  depends  on  the  ability 
to  attract  and  retain  outstanding  researchers  in  the  Air  Force  laboratories.  In 
addition  to  the  research  they  produce,  in-house  staff  provide  an  essential 
communication  channel  and  means  of  collaborating  with  external  researchers. 

Air  Force  salaries  and  other  benefits  are  somewhat  lower  than  those  of 
comparable  industrial  and  university  laboratories.  Efforts  to  increase  Air 
Force  salaries  should  continue.  However,  in  discussions  with  laboratory 
scientists,  the  subcommittee  learned  that  many  find  other  aspects  of  Air  Force 
research  programs— the  excellent  facilities  and  equipment  and  fewer 
requirements  to  write  lengthy  research  proposals— to  outweigh  the  salary 
deficit. 

A  related  issue  is  how  to  bring  talented  new  staff  into  the  Air  Force 
laboratories.  The  new  Palace-Knight  program,  to  increase  young  doctoral-level 
scientists  and  engineers  in  the  laboratories,  appears  to  be  an  important  step 
taken  to  attract  new  staff.  However,  this  program  is  limited  in  numbers  and 
the  approach  of  identifying  people  so  early  in  their  academic  careers  has  not 
yet  stood  the  test  of  time.  Discussions  with  current  young  Air  Force  staff 
suggested  that  they  are  attracted  by  the  same  benefits  that  help  retain  the 
older  staff. 


Conclusions 

Cutbacks  in  funding  and  the  Air  Force  work  force  will  make  it 
increasingly  difficult  for  Air  Force  laboratories  to  maintain  an  inflow  of  good 
young  staff.  In  view  of  the  importance  of  the  issue,  ways  to  overcome  any  such 
problems  must  be  found.  The  Air  Force  should  consider  how  to  maintain 
state-of-the-art  experimental  facilities  in  the  laboratories.  With  current  funding 
cuts,  there  is  a  concern  that  these  facilities  could  rapidly  deteriorate.  Not  only 
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would  this  seriously  hamper  lab  research,  it  would  make  it  more  difficult  to 
hire  top  researchers. 

Finally,  care  must  be  given  not  to  overburden  laboratory  staff  with 
administrative  work  and  with  proposal  writing.  That  is  not  to  say  that  proposal 
writing  should  be  eliminated  or  that  service  on  committees  and  task  forces 
should  be  stopped.  However,  an  appropriate  balance  must  be  struck.  As 
suggested  earlier,  the  Air  Force  should  encourage  more  planning  and 
management  input  from  the  laboratories,  but  at  the  same  time  not  overburden 
productive  scientists  with  administrative  matters.  Air  Force  administration  and 
the  chief  scientists  at  the  laboratories  could  help  ensure  that  adequate  support 
staff  is  available  to  avoid  undue  burdens  on  research  staff. 
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ATTACHMENT  A-1 

Guiding  Principles 
for 

The  Intramural  Basic  Research  Program  Sponsored  bv  AFQSR 


1.  Mission:  The  Air  Force  Office  of  Scientific  Research  (AFOSR)  has  single 
manager  responsibility  for  planning,  managing,  implementing,  and  controlling 
the  USAF  Defense  Research  Sciences  program  under  Program  Element 
61102F.  In  that  role,  AFOSR  executes  the  policy  prescribed  by  AFSC 
Regulation  23-15, 17  June  1988.  In  addition,  AFOSR  shall  work  closely  with 
the  Air  Force  laboratories  to  nurture  and  support  quality  research  and  to 
coordinate  intramural  and  external  research  results  to  promote  transition  of 
research  to  the  Exploratory  Development  programs  of  the  laboratories. 

2.  Purpose  of  Intramural  Tasks:  To  perform  research;  contribute  to  the 
knowledge  base;  augment  technology  transition;  provide  and  develop  expertise 
in  fundamental  S&E;  attract  and  develop  talent;  and  help  assure  relevancy  of 
the  extramural  program. 

3.  Scope  of  Intramural  Program:  The  intramural  program  is  defined  as 
funded  tasks  in  those  Air  Force  organizations  that  fall  under  the  management 
guidance  of  the  Technology  Executive  Officer  (TEO);  currently  this  includes 
WL,  AL,  PL,  RL  and  AFCESA/RA.  The  total  investment  should  be 
approximately  one  third  of  the  discretionary  portion  of  the  6.1  DRS  funds. 
ITie  discretionary  portion  is  arrived  at  by  subtracting  AFOSR  support  funds 
(including  those  for  detachments),  withholds  by  parent  organizations, 
Congressionally  mandated  specific  programs,  etc.  The  detailed  distribution 
among  the  laboratories,  their  directorates,  and  other  organizations  may 
substantially  deviate  firom  the  overall  one-third  goal;  the  detailed  distribution 
is  governed  by  technical  and  management  considerations  aimed  at  optimizing 
overall  benefits  to  the  Air  Force. 

4.  Execution  of  Intramural  Program:  The  intramural  program  is  subdivided 
into  laboratory  tasks  each  headed  by  a  task  manager.  To  achieve  its  purpose, 
the  intramural  program  must,  primarily,  be  carried  out  by  Government 
employees  within  the  laboratories  of  their  employer.  Visits  and  working  stays 
by  others  on  the  laboratory  premises  as  well  as  by  Government  employees 
elsewhere  are  encouraged.  To  the  degree  necessitated  by  hiring  constraints, 
on-site  contractor  employees  are  acceptable.  Extramural  contracts  or  grants 
(research  performed  by  others  off  premises,  foreign  or  domestic)  are  to  be 
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used  only  to  directly  support  or  complement  (as  part  of  a  coordinated 
program)  on-going  in-house  research  programs.  This  includes  collaborative 
work  wiA  external  researchers. 

5.  Laboratory  Interface: 

a.  The  AFOSR  program  manager  (PM)  is  the  primary  interface  to  the 
laboratory  tasks  within  a  PM’s  "portfolio"  (AFOSR  subarea).  Funding 
decisions,  annual  reporting,  initiation  of  research,  guidance  of  research  etc., 
are  made  at  this  level.  A  PM  works  with  laboratory  staff,  laboratory  task 
managers,  division  chiefs  and  the  directorate  chief  scientists. 

b.  AFOSR  scientific  directors  are  the  higher  level  interface.  Decisions 
on  funding  shifts  between  AFOSR  subareas  and  decisions  involving  policy 
matters,  as  well  as  on  the  quality  of  the  program,  will  be  made  at  this  level. 
Scientific  directors  will  typically  interact  with  the  directorate  directors, 
directorate  chief  scientists  and  the  laboratory  chief  scientists. 

c.  To  aid  in  strategic  research  planning,  decision  making,  vectoring  of 
limited  resources,  and  arbitration  of  differences  in  priorities,  two  advisory 
boards  have  been  formed:  the  Technical  Advisory  Boards  (TABs),  providing 
advice  to  AFOSR  scientific  directors,  and  the  AFOSR  Research  Council, 
providing  advice  to  the  director  of  AFOSR. 

6.  External  Interface:  To  promote  technology  transition  from  6.1  to 
laboratory  6.2  programs,  the  AFOSR  program  managers  and  scientific 
directors  will  (1)  promote  collaborative  programs  between  laboratory  scientists 
and  external  researchers,  (2)  promote  visits  and  scientific  exchange  between 
university  and  laboratory  scientists,  and  (3)  sponsor,  at  least  armually,  joint 
AFbSR-Air  Force  laboratory  topical  worlshops  for  the  exchange  of  technical 
information  between  Air  Force  laboratory  researchers  and  external 
researchers  funded  directly  by  AFOSR;  such  workshops  may  be  topically 
focused  and  may  be  open  to  others  such  as  to  the  other  Services. 

7.  Frmding  Policy: 

a.  Research  in  Air  Force  laboratories  will  be  initiated  based  on  a  written 
request  which  will  be  evaluated  by  AFOSR;  progress  will  be  reported  armually 
(See  AFOSRR  80-7).  Substantial  changes  in  the  scope  or  direction  of 
research  will  be  considered  equivalent  to  an  initiation  of  new  research 
requiring  appropriate  documentation. 
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b.  There  is  no  automatic  termination  of  intramural  research.  K  a 
laboratory  task  must  be  terminated  for  reasons  other  than  natural  completion 
or  personnel  departure,  coordination  for  termination  should  be  accomplished 
through  the  appropriate  AFOSR  TAB(s)  and,  if  required,  through  the 
AFOSR  Research  Cbuncil. 

c.  Funding  may  be  handled  through  a  Budget  Authority  (BA)  or  other 
funding  instrument  as  authorized  by  the  USAF. 

d.  AFOSR  generally  will  accept  overhead  charges  by  the  Air  Force 
laboratories  associated  with  management  and  support  of  research;  however, 
unreasonable  additional  withholds,  not  reconcilable  with  carrying  out  basic 
research,  may  be  grounds  for  AFOSR  to  not  fund  intramural  research. 

8.  Reviewing  Research: 

a.  The  responsibility  for  ascertaining  excellence  of  research  and  its 
relevance  is  delegated  by  the  Air  Force  to  AFOSR.  In  carrying  out  its 
management  duties,  AFOSR  must  rely  on  competent  counsel.  For  relevance, 
this  counsel  primarily  comes  from  senior  Air  Force  laboratory  staff. 
Excellence  is  determined  by  the  AFOSR  scientific  directors,  augmented  by 
additional  peer  reviews  as  necessary. 

b.  AFOSR  will  schedule  two  oversight  processes  each  year.  The  Fall 
Reviews  are  for  the  purpose  of  evEiluating  the  entire  basic  research  program 
for  excellence  and  relevance.  The  Spring  Reviews  deal  with  programmatic 
direction  and  planning,  and  include  the  selection  of  new  programmatic 
initiatives  for  fonding.  The  AFOSR  Research  Council  and  the  Technical 
Advisory  Boards  participate  in  these  processes. 

9.  This  document  supersedes  the  following  MOUs: 

a.  MOA  between  AFOSR  and  the  Air  Force  Engineering  and  Services 
Laboratory,  15  August  1988 

b.  MOA  between  AFOSR  and  the  Air  Force  Space  Technology  Center, 
11  April  1989 

c.  MOA  between  AFOSR  and  the  Human  Systems  Division,  27 
September  1988 
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d.  MOA  between  AFOSR  and  Rome  Air  Development  Center,  27 
September  1988 

f.  MOA  between  AFOSR  and  the  Wright  Research  and  Development 
Center,  4  October  1989 

g.  MOA  between  AFOSR  and  the  Air  Force  Armament  Laboratory,  30 
August  1988 


HELMUT  HELLWIG 
Director 


CONCURRENCE: 


AL/CA  (For  Armstrong  Laboratory) 


PL/CA  (For  Phillips  Laboratory) 

RL/CA  (For  Rome  Laboraton^^^~7!:^^ 
WL/CA  (For  Wright  Laboratory)  M 
AFCESA/RA  ^~yy\ulL^  Jt), 


DATE: 

DATE: 

DATE:  /« 
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Appendix  B: 

A  Comparison  of  Research  Management  Practices 
at  the  Air  Force  Office  of  Scientific  Research 
and  Other  DOD  and  Federal  Agencies 

Paul  F.  Parks,  Chairman 
Albert  J.  Baciocco 
Erich  P.  Ippen 
Christopher  C.  Green 

The  subcommittee  was  charged  with  comparing  the  Air  Force’s  research 
management  practices  and  those  of  other  government  organizations,  to 
determine  what  lessons  these  practices  might  offer  for  Air  Force  management 
of  basic  (6.1)  research.  The  subcommittee  was  also  asked  to  consider,  as 
possible,  related  issues  of  research  leveraging  and  redundancy. 

In  conducting  its  study,  the  subcommittee  reviewed  the  research 
management  practices  of  both  Department  of  Defense  (DOD)  and  other 
federal  agencies.  Members  of  the  subcommittee  were  briefed  by  the  director, 
program  managers,  or  other  senior  managers  of  the  Air  Force  Office  of 
Scientific  Research  (AFOSR),  Army  Research  Office  (ARO),  Office  of  Naval 
Research  (ONR),  Office  of  the  Secretary  of  Defense  (OSD),  Defense 
Advanced  Research  Projects  Agency  (DARPA),  Air  Force  laboratories.  Air 
Force  Scientific  Advisory  Board  (SAB),  National  Science  Foundation  (NSF), 
U.S.  Department  of  Energy  (DOE),  and  National  Aeronautics  and  Space 
Administration  (NASA)  (see  Appendix  D  for  a  list  of  these  meetings). 


DOD  AGENCIES 

The  briefings  on  AFOSR,  ARO,  ONR,  and  DARPA  led  the 
subcommittee  to  focus  especially  on  several  research  management  issues  in 
the  context  of  DOD  organizations:  the  reporting  level  of  the  research 
organization  within  the  greater  DOD  organization;  6.1  resource  management 
issues,  including  the  level  and  allocation  of  6.1  funding,  research  management 
persormel,  and  discretionary  laboratory  funding;  and  interservice  cooperative 
research. 
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Reporting  Level  of  the  Research  Organization 

In  response  to  recent  budgetary  and  geopolitical  trends— and  to  many 
recent  studies  of  defense  research  management— all  three  military  services 
have  been  reorganizing  their  laboratories  and  R&D-related  processes  over  the 
past  few  years.^ 


Army 


The  Army  has  combined  many  of  its  laboratories  and  changed  its  lab 
command  structure.  Under  the  new  structure,  the  Army  Materiel  Command’s 
(AMC)  Laboratory  Command  (LABCOM)  and  its  seven  corporate 
laboratories  were  consolidated  in  the  new  Army  Research  Laboratory  (ARL). 
The  Director  of  ARO,  which  manages  extramural  Army  research,  no  longer 
reports  through  LABCOM,  but  instead  directly  to  AMC  Headquarters  (Figure 
B-1  shows  the  new  Army  organizational  structure).  The  Army  also  continues 
to  maintain  laboratories  in  the  Army  Medical  Research  and  Development 
Command  and  Army  Corps  of  Engineers,  and  conducts  additional  research 
through  its  Research,  Development,  and  Engineering  Centers  (RDECs). 

An  important  advantage  of  these  organizational  changes,  beyond  the 
greater  integration  and  efficiency  they  promote,  is  that  basic  research 
opportunities  and  needs,  especially  those  relating  to  extramural  research,  are 
given  greater  visibility  withio  the  Army  structure.  The  Commanding  General 
of  AMC  reports  directly  to  the  Chief  of  Staff  of  the  Army  and  coordinates  6.1 
programs  with  the  Assistant  Secretary  of  the  Army  for  Research, 
Development  and  Acquisition.  The  Director  of  ARO  is  now  also  a  voting 
member  of  the  AMC  Science  and  Technology  Board  of  Directors,  which 
coordinates  basic  research  and  related  activities,  and  whose  other  members 
include  the  directors  of  ARL  and  the  RDECs,  as  well  as  the  AMC  Chief 
Scientist  and  representatives  of  the  Department  of  the  Army  and  Army 
Training  and  Doctrine  Command. 


^  For  an  overview  of  these  developments,  see  Federal  Advisory  Commission  on 
Consolidation  and  Conversion  of  Defense  Research  and  Development  Laboratories,  Report  to 
the  Seaetary  of  Defense,  September  1991.  Other  important  recent  reports  include  the  Defense 
Science  Board  1987  Summer  Study  on  Technology  Base  Management,  December,  1987;  and 
U.S.  Cot^ess,  Office  of  Technology  Assessment,  Holding  the  Edge:  Maintaining  the  Defense 
Technology  Base,  U.S.  Government  Printing  Office,  June  1989. 
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FIGURE  B-1  Army  organization  for  research  management  (adapted  from  Army  Science  and  Technology  Master 
Plan.  Headquarters,  Department  of  the  Army,  Noyember  1992;  solid  lines  are  reporting  lines,  dotted  lines  are  lines 
of  coordination;  for  explanation  of  acronyms,  see  glossary). 
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Navy 


The  recent  Navy  Laboratory  Consolidation  Plan  was  developed  to 
consolidate,  realign,  and  downsize  the  Navy’s  R&D  centers,  merging  them 
with  support  engineering  activities  and  T&E  centers,  to  form  four  Navy 
warfare  centers.  The  two  laboratories  of  the  Office  of  Naval  Research  (ONR), 
which  is  responsible  for  6.1  programs,  have  also  been  consolidated:  the  Naval 
Research  Laboratory  (NRL)  and  the  Naval  Oceanographic  and  Atmospheric 
Research  Laboratory  (NOARL)  were  combined  at  the  beginning  of  fiscal  year 
1993  to  form  a  new  NRL. 

In  keeping  with  long-established  practice,  the  Chief  of  Naval  Research 
will  continue  to  be  responsible  for  the  planning  and  execution  of  the  entire 
Department  of  the  Navy  science  and  technology  enterprise,  including  basic 
research  through  the  ONR,  This  position  has  no  direct  counterpart  in  the 
other  services  (Figure  B-2  shows  the  Navy  research  organizational  structure). 
Along  with  other  characteristics  of  the  Navy  research  enterprise,  this  position 
has  encouraged  appropriately  high-level  attention  to  basic  research.  The  Chief 
of  Naval  Research  reports  directly  to  the  Assistant  Secretary  of  the  Navy  for 
Research,  Development  and  Acquisition. 


^ir  Force 

The  Air  Force  has  just  undergone  a  major  realignment  of  its  laboratories 
and  research  management.  The  previous  14  Air  Force  laboratories  were 
combined  into  4  "super"  laboratories:  Wright  (aircraft),  Phillips  (space  and 
missiles),  Armstrong  (human  systems),  and  Rome  (command,  control, 
communications,  and  intelligence,  or  "C?!")  laboratories.  This  realignment 
reflects,  the  move  from  a  technology-oriented  to  a  systems-oriented, 
multidisciplinary  laboratory  approach,  and  is  designed  to  promote  efficiency. 

Effective  July  1,  1992,  the  management  structure  under  which  AFOSR 
operates  has  also  been  changed.  Air  Force  Systems  Command  (AFSC),  to 
which  AFOSR  reported,  was  combined  with  Air  Force  Logistics  Command 
(AFLC),  to  form  the  new  Air  Force  Materiel  Command  (AFMC). 

Under  the  previous  Air  Force  organization,  the  AFOSR  Director 
reported  to  the  AFSC  Deputy  Chief  of  Staff  for  Technology,  who  reported  to 
the  AFSC  Commander.  The  Deputy  Chief  of  Staff,  as  Technology  Executive 
Officer  (TEO),  also  reported  to  the  Assistant  Secretary  of  the  Air  Force  for 
Acquisition.  The  advantage  of  the  TEO  role  is  in  providing  integrated  S&T 
investment  strategy  guidance  and  a  dedicated  lab  systems  advocate.  These 
particular  features  of  the  Air  Force  command  structure  remain  fundamentally 
imchanged  (Figure  B-3  shows  the  new  Ar  Force  structure). 


FIGURE  B-2  Navy  organization  for  research  management  (adapted  from  material  provided  by  the  Office  of  Naval 
Research). 


FIGURE  B-3  Air  Force  organization  for  research  management  (Headquarters,  Air  Force  Materiel  Command). 
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However,  with  the  formation  of  AFMC,  AFOSR  has  been  placed  within 
a  much  larger,  more  diverse  organization.  In  1990,  AFSC  personnel  totaled 
about  44,000;  in  1992  (following  the  Air  Force  reorganization),  AFMC 
personnel  totaled  almost  three  times  that,  or  nearly  129,000.^  Representatives 
of  both  the  Air  Force  Scientific  Advisory  Board  and  AFOSR,  as  well  as  the 
subcommittee,  have  some  concerns  about  the  effect  of  this  change  on 
AFOSR’s  ability  to  manage  the  Air  Force  basic  research  programs. 
Moreover,  AFOSR  is  now  reporting  to  a  one-star  general  under  AFMC,  while 
under  AFSC  it  reported  to  a  two-star  general;  ONR  reports  to  an  admiral  of 
two-star  rank,  who  reports  directly  to  the  Office  of  the  Secretary  of  the  Navy; 
and  ARO  was  recentiy  "upgraded"  to  three-star  reporting.  These  differences 
in  reporting  level  are  not  of  concern  in  and  of  themselves.  However,  reporting 
level  in  part  determines  the  visibility  of  any  DOD  organization  within  DOD 
(e.g.,  in  assessing  the  relative  importance  of  programs),  and  thus  also  its 
visibility  externally,  that  is,  academia  and  industry  about  the  organization’s 
standing. 

The  subcommittee  is  concerned  about  the  effects  of  the  Air  Force 
reorganization  on  AFOSR,  and  especially  about  the  combining  of  AFSC  and 
AFLC  to  form  AFMC.  This  reorganization  places  AFOSR  in  a  much  larger, 
more  diverse  organization  where  it  may  be  further  removed  fi'om  senior 
leadership,  with  a  resulting  loss  of  critical  coimection  to  planning  and  policy. 
The  post-Cold  War  world  presents  fewer  military  challenges  in  important 
respects,  but  the  importance  of  basic  research  to  the  Air  Force  mission  is  still 
as  vital.  There  is  growing  worldwide  availability  of  high  technology,  use  of 
such  technology  in  weapons  systems,  and  unpredictability  of  threats  to  the 
national  interest. 


Defense  Advanced  Research  Projects  Agency 

The  subcommittee  also  looked  at  DARPA,  to  see  what  lessons  it  might 
offer  for  organizational  structure.  However,  in  this  and  other  points  of 
comparison,  the  subcommittee  found  that  DARPA’s  specialized  focus  (mainly 
high-risk  demonstration  and  fieldable  advanced  technical  systems)  and  related 
operations  fail  to  provide  any  readily  determinable  lessons  for  Air  Force  basic 
research  management. 


Source  of  figures,  AFMC  Headquarters. 
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Conclusions 

The  importance  of  basic  research  to  the  Air  Force’s  long-term 
warfighting  capabilities  must  be  recognized  by  its  organizational  structure.  For 
this  reason,  it  is  critical  that  the  Air  Force  basic  research  organization  be 
positioned  to  inform  and  be  informed  by  the  policy,  budget,  and  program 
decisions  that  may  affect  the  contributions  of  basic  research  to  the  .Adr  Force 
mission.  Within  the  current  structure,  this  goal  might  be  best  accomplished  by 
ensuring  the  TEO  maintains  his  current  level  of  responsibility  and  role,  and 
by  continuing  to  assign  TEO  responsibilities  to  a  person  who,  by  virtue  of 
training  and  background,  demonstrates  strong  knowledge  and  appreciation  of 
basic  research. 


Management  of  Basic  Research  (6.1)  Resources 

The  Army  Research  Office  manages  only  the  Army’s  extramural  basic 
research  programs,  including  its  Centers  of  Excellence  (e.g.,  in  electronics  and 
rotorcraft),  which  consist  mainly  of  university  research.  Thus,  ARO  manages 
only  about  one-third  of  the  Army’s  6.1  funds.  The  other  two-thirds  are 
allocated  directly  to  the  Army  laboratories  (primarily  by  Army  Materiel 
Command,  but  also  by  other  commands),  including  ffinding  for  In-House 
Laboratory  Independent  Research  (lUR).  ILIR  funds  are  lab  discretionary 
funds,  to  pursue  projects  of  high  risk,  but  high  potential  payoff,  and  irmovative 
approaches  to  research.  A  significant  fraction  of  the  funding  allocated  directly 
to  Army  laboratories  also  goes  out  to  support  university  and  industry  research. 

In  contrast  to  ARO,  the  Office  of  Naval  Research  manages  all  Navy  6.1 
funding,  both  intra-  and  extramural,  that  is,  allocations  to  universities,  industry 
and  other  government,  and  Navy  laboratories,  including  ILIR-funded  work. 
For  fiscal  year  1992,  the  breakdown  of  Navy  research  performers,  by 
percentage  of  the  ONR  budget,  was  as  follows:  universities,  58  percent;  Navy 
laboratories,  30  percent;  and  industry  and  other  government,  12  percent.^ 
(Figures  for  1990  and  1991  were  also  close  to  these  values.) 

In  the  Air  Force,  AFOSR  also  manages  all  6.1  funds,  with  the  exception 
of  IT  JR  funds,  which  are  managed  by  the  laboratories,  d^urrent  AFOSR 
guidance  suggests  that  about  one-third  of  the  Air  Force  basic  research 
(Defense  Research  Sciences)  funding  should  go  to  the  intramural  program. 
(See  Attachment  A-1  for  the  full  text  of  this  guidance  document,  which  also 
spells  out  specific  roles  for  the  management  of  Air  Force  intramural  basic 


^  Source  of  figures.  Office  of  the  Chief  of  Naval  Research. 
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research.)  As  in  the  other  services,  Air  Force  basic  research  funds  that  are 
allocated  to  the  laboratories  are  sometimes  used  to  support  outside  work. 

The  three  services’  different  ways  of  distributing  6.1  funds  each  have 
their  own  advantages.  Some  are  noted  in  the  discussion  that  follows.  However, 
the  subcommittee  saw  no  reason  to  question  the  general  Air  Force  practice 
in  this  area,  that  is,  AFOSR’s  designation  as  "the  single  manager  for  the 
Defense  Research  Sciences  program"  (e.g.,  in  contrast  to  Army  practice). 

According  to  information  provided  by  DOD  and  the  services’  offices  of 
research.  Army  6.1  funding  during  fiscal  year  1992  is  approximately  $197 
million.  Navy  6.1  funding  $422  million,  and  Air  Force  6.1  funding  $203  million 
(Figure  B-4).  These  values  represent  3.6  percent  of  the  Army’s  total  Research, 
Development,  Test  and  Evjduation  (RDT&E)  funding,  4.2  percent  of  the 
Navy’s;  and  1.5  percent  of  the  Air  Force’s  (Figure  B-5).  Thus,  the  Air  Force 
appears  to  commit  a  significantly  smaller  percentage  of  its  total  RDT&E 
funds  to  basic  research  than  either  of  the  other  two  services.  The 
subcommittee  was  not  briefed  on  the  reasons  for  this  discrepancy,  nor  was  it 
able  to  investigate  this  complex  subject  in  satisfactory  depth.  Further  study  of 
the  issue  may  be  in  order. 

As  Figure  B-4  shows,  DOD  funding  for  basic  research  has  been 
reasonably  stable  in  constant  dollars  over  the  past  decade.  Stability  of  funding 
is  critical  for  basic  research,  in  which  important  results  are  generally  less 
predictable  and  seen  over  a  longer  term  than  in  applied  research.  Briefers  to 
the  committee  indicated  that  the  Air  Force  is  attempting  to  maintain  the 
stability  of  basic  research  funding,  even  in  the  face  of  likely  severe  budget  cuts 
over  the  next  several  years.  This  goal  is  both  desirable  and  practical,  if  the 
benefits  of  basic  research  are  to  be  achieved. 

At  the  same  time,  the  subcommittee  is  concerned  that,  as  often  occurs 
in  such  cases,  budgetary  trends  have  increased  the  pressures  to  allocate  a 
greater  fi-action  of  Air  Force  basic  research  funding  to  in-house  laboratories. 
Strong  intramural  basic  research  is  essential  for  mission-specific  work.  But  an 
equally  or  more  important  function  of  Air  Force  intramural  research  is  to 
help  Air  Force  laboratories  absorb  the  far  more  extensive  research  advances 
in  universities  and  industry. 

The  Air  Force  must  continue  to  capitalize  well  on  the  much  greater  body 
of  outside  research.  Moreover,  extramural  research  is  generally  highly 
leveraged,  financially  and  intellectually.  In  the  case  of  university  research,  it 
additionally  prepares  the  S&T  work  force  on  which  the  future  Air  Force 
depends.  For  these  reasons,  the  subcommittee  would  not  support  any  change 
in  policy  to  direct  a  greater  fraction  of  Air  Force  basic  research  funding  to 
intramural  research.  In  reviewing  AFOSR’s  competitive  processes,  the 
subcommittee  noted  that  the  Air  Force  laboratories  provide  important 
feedback  about  the  results  of  AFOSR-sponsored  research  (as  users  and  raters 
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FIGURE  B-4  Service  basic  research  budgets,  in  constant  1990  dollars  (DOD 
Office  of  Research  and  Laboratory  Management). 
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NAVY 


FIGURE  B-5  Service  basic  research 
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of  research  results,  and  as  enablers  of  technology  transfer).  At  the  same  time, 
the  laboratories  receive  basic  research  funding  from  AFOSR.  Thus,  to  avoid 
conflicts  of  interest,  care  should  be  taken  in  such  arrangements  to  ensure  that 
the  laboratories  do  not  exercise  undue  influence  over  allocations  of  Air  Force 
6.1  funding. 

While  the  primary  driving  force  for  research  in  the  laboratories  and 
extramural  programs  should  be  long-term  Air  Force  needs,  as  formulated  by 
Air  Force  leadership,  the  subcommittee  also  supports  the  recommendation  of 
the  1983  Federal  Laboratory  Review  Panel  (reaffirmed  by  the  1987  Defense 
Science  Board  study  Technology  Base  Management,  earlier  cited)  that  "at 
least  5%,  and  up  to  10%,  of  the  annual  funding  of  the  federal  laboratories 
should  be  devoted  to  a  program  of  independent  research  and  development  at 
the  laboratory  director’s  discretion."  According  to  such  recent  studies,  and  also 
in  keeping  with  observations  to  the  subcommittee  and  its  own  experience,  this 
management  flexibility  provides  the  basis  for  much  creative  work. 
Discretionary  funds  are  important,  as  the  DSB  study  observed,  to  maintain  the 
vitality  of  the  laboratories.  lUR  funds  are  one  important  kind  of  such 
discretionary  funds  and  should  be  protected.  At  the  same  time,  visibility  on 
this  work  must  be  afforded  to  the  rest  of  the  Air  Force  research  community, 
in  particular  to  AFOSR. 

Air  Force  laboratories  often  contract  with  local  universities  for 
analytical,  engineering,  and  other  services  using  basic  research  dollars.  The 
subcommittee  is  concerned  because  these  funds  are  not  competitively 
awarded.  In  fact,  based  on  the  information  provided,  two  issues  arise  about 
the  current  practice:  are  these  research  dollars  well  used,  and  do  they  support 
what  is  properly  considered  basic  research?  The  subcommittee  believes  such 
funds  should  be  awarded  through  appropriate  competitive  processes  and  as 
6.1  dollars. 

Good  basic  research  programs  require  highly  skilled  and  creative 
managers,  who  hold  their  positions  long  enough  to  see  research  programs 
through.  This  requirement  is  stronger  in  the  case  of  Air  Force  basic  research 
than  in  other  environments,  in  that  Air  Force  basic  research  managers  have 
great  technical  and  decision-making  authority.  The  program  manager  is  the 
main  AFOSR  interface  to  laboratory  tasks  within  a  given  technical  area,  and 
has  primary  responsibility  for  basic  research  funding  decisions,  annual 
reporting,  and  initiation  and  guidance  of  research  (again,  see  Attachment  A- 
!)• 

It  has  long  been  recogmzed  throughout  DOD  that  the  need  for  highly 
qualified,  stable  technical  leadership  is  difficult  to  achieve  through  the 
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rotation  of  military  officers/  Moreover,  the  committee  also  heard  some 
reports  that  the  Air  Force  fails  to  look  on  research  management  assignments 
as  positive  career  moves  (though  this  view  was  not  universal). 

An  unwritten  policy  at  AFOSR  is  that  each  directorate  should  have  two 
military  program  managers;  together,  they  represent  25  to  30  percent  of  all 
AFOSR  program  managers.  AFOSR  reports  no  significant  difference  in 
performance  between  its  military  and  civilian  research  managers.  Other 
observers  have  also  pointed  out  that  military  research  managers  provide  useful 
perspective  on  Air  Force  field  needs. 

However,  based  on  all  available  evidence,  the  subcommittee  believes 
that  the  more  effective  way  to  ensure  program  quality  and  stability  and 
organizational  memory  is  generally  through  the  recruitment  of  distinguished 
civilian  leaders  as  basic  research  managers.  Longer  tours  of  duty  might  also 
be  considered  for  military  research  managers,  and  their  career  opportunities 
should  be  good.  Program  managers  should  have  adequate  training  and 
experience  (including  technical  management  training  and  3  to  10  years  of 
technical  management  experience).  Above  all,  the  Air  Force  should  continue 
to  guarantee  that  its  research  managers  demonstrate  the  high-quality  technical 
leadership  that  AFOSR  now  attributes  to  them. 


Conclusions 

The  subcommittee  reached  the  following  conclusions  based  on  its 
comparative  review  of  6.1  resource  management: 

1.  The  subcommittee  strongly  supports  the  Air  Force  policy  of 
maintaining  6.1  funding  at  no  less  than  its  current  level  adjusted  for  inflation. 
Stability  of  funding  is  essential  to  the  welfare  of  basic  research  programs, 
which,  by  their  nature,  are  long-term  commitments.  Funding  stability  is  also 
critical  to  maintain  Air  Force  core  competencies. 

2.  The  subcommittee  does  not  support  of  any  change  of  policy  to  direct 
a  greater  portion  of  6.1  funding  to  the  laboratories.  While  intramuraJ  research 
plays  its  own  critical  roles,  extramural  basic  research  represents  a  much  more 
extensive  body  of  scientific  expertise.  It  is  also  usually  highly  leveraged,  in 
funds,  equipment,  and  intellectual  resources.  Additionally,  university  research 
prepares  the  future  Air  Force  scientific  work  force  through  the  support  of 
graduate  and  postdoctoral  programs.  Beyond  taking  these  factors  into  account 
in  its  allocations  to  intra-  and  extramural  programs,  the  Air  Force  should 


^  For  related  discussion,  see  the  1991  Report  to  the  Secretary  of  Defense  and  1987  Defense 
Science  Board  report  previously  cited. 
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distribute  its  6.1  funds  based  simply  on  the  value  of  the  research  to  the  needs 
of  the  Air  Force  technology  base,  the  quality  of  the  research,  and  the  quality 
of  the  researchers. 

3.  In  general,  the  Air  Force  should  ensure  program  quality  and  stability 
and  organizational  memory  by  emphasizing  the  recruitment  of  experienced, 
technically  proficient,  civilian  leaders  as  managers  of  its  basic  research 
programs.  For  the  same  reason,  longer  tours  of  duty  should  be  considered  for 
military  research  managers,  and  they  should  be  assured  good  career 
opportunities.  All  research  managers  should  have  strong  skills  and  experience 
in  technical  management  (including  technical  management  training  and  3  to 
10  years  of  technical  management  experience). 

4.  The  subcommittee  strongly  supports  the  aimual  budgeting  of 
independent  research  and  development  funds,  to  be  used  at  the  discretion  of 
the  Air  Force  laboratory  directors.  This  funding  provides  the  flexibility  that 
allows  the  development  of  innovative  ideas.  Such  discretionary  funding, 
including  HJR  funding,  should  be  protected.  At  the  same  time,  AFOSR 
should  be  given  review  responsibility  for  ILIR  work,  to  help  provide  a 
common  standard  for  quality  and  to  assist  in  the  integrated  planning  of  Air 
Force  research,  especially  to  avoid  research  redundancy. 


Cooperative  Research 

The  Air  Force’s  effective  management  of  basic  research  depends 
significantly  on  the  degree  to  which  it  capitalizes  on  the  work  of  others, 
including  that  of  the  other  military  services.  For  this  reason,  the  subcommittee 
also  looked  at  some  of  the  interservice  cooperative  research  in  which  the  Air 
Force  is  participating. 

In  October  1989,  Deputy  Secretary  of  Defense  Donald  Atwood 
challenged  the  services  to  create  new  approaches  to  S&T  management,  to 
increase  efficiency  and  reduce  unwarranted  overlap  in  the  RDT&E  activities 
of  the  military  departments.  This  challenge  led  to  the  Tri-Service  S&T 
Reliance  project,  as  well  as  to  the  laboratory  consolidation  studies  for  the 
three  services  whose  results  were  reviewed  above.^ 

Tri-Service  S&T  Reliance  issued  from  Project  Reliance,  a  joint  study  of 
the  Army  and  the  Air  Force  on  the  consolidation  and  collocation  of  their 
R&D  efforts  in  selected  technology  areas.  Tri-Service  S&T  Reliance  expanded 


^  For  more  background  on  the  Reliance  Project,  see  Joint  Directors  of  Laboratories  (JDL), 
"Tri-Service  Reliance  in  Science  &  Technology,"  White  Paper,  January  1992;  and  JDL,  Tri- 
Service  Science  &  Technology  Reliance:  Annual  Report,  December  1992. 
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this  approach  to  include  the  Navy,  initiating  some  of  the  most  significant 
restructuring  of  the  technology  base  in  the  past  40  years. 

Tri-Service  S&T  Reliance  consists  of  a  set  of  formal  agreements  among 
and  implemented  by  the  military  departments  for  joint  planning,  collocated 
in-house  work,  and  lead-service  assignment,  covering  most  non-service-unique 
6.1,  6.2,  and  6.3 A  programs.  Reliance  is  also  a  formal  process,  authorized  by 
each  service  Acquisition  Executive  and  approved  by  the  Deputy  Secretary  of 
Defense,  to  streamline  the  S&T  programs  of  the  military  departments  and 
better  plan  the  national  defense  S&T  investment.  Responsibility  for 
implementing  Reliance  has  been  assigned  to  the  Joint  Directors  of 
Laboratories  (JDL). 

Beyond  reducing  unwarranted  duplication  and  increasing  efficiency,  goals 
of  Reliance  include  ensuring  a  critical  mass  of  resources  to  develop 
world-class  products,  and  preserving  mission-essential  capabilities. 

Six  categories  of  interservice  reliance  were  envisioned:  coordination, 
joint  efforts,  collocation,  consolidation,  competition,  and  service-unique 
work.®  Figure  B-6  presents  the  technology  areas  addressed  by  working  groups 
during  the  study  phase  of  this  project,  and  the  oversight  bodies  of  these  areas. 
The  study  phase  of  project  Reliance  resulted  in  formal  service  agreements  for 
joint  planning,  collocated  research,  or  consolidation  under  a  lead  service  for 
each  technology  area  that  is  not  service-unique. 

The  results  of  these  agreements  were  as  follows: 

•  In  71  technology  areas/subareas/sub-subareas,  the  services  will 
jointly  plan  the  work  to  be  conducted  at  separate  service  locations,  as 
contrasted  with  the  6  such  cases  previously. 

•  In  105  technology  areas/subareas/sub-subareas,  work  will  be 
collocated  to  various  single-service  sites  for  program  execution,  as  contrasted 
with  the  13  such  cases  earlier. 

•  Service  management  leads  will  be  designated  for  10  technology 
areas/subareas/sub-subareas,  an  increase  of  1  over  the  past. 


®  Coordination  has  traditionally  been  the  most  common  type  of  interaction  among  the 
services,  represented  by  the  many  coordination  bodies  developed  over  the  decades.  Joint  efforts 
are  planned  and  conducted  jointly,  but  the  involved  services  can  execute  tasks  at  separate 
locations  and  they  retain  separate  funding  control.  Collocation  is  in-house  task  execution 
throu^  a  single  service,  but  with  all  services  maintaining  separate  funding  control.  In 
consolidation,  a  lead  service  manages  the  program  and  all  related  S&T  funds  are  transferred 
to  the  lead  service.  Competition  refers  to  in-house  task  execution  that  is  competed  among  the 
service  performers,  with  all  services  retaining  funding  and  performer-decision  control.  Service- 
unique  work  is  that  work  peculiar  to  a  given  service,  for  which  no  interservice  reliance  is 
required  or  appropriate. 
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JOINT  DIRECTORS  OF  LABORATORIES 
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FIGURE  B-6  Technology  areas  addressed  by  project  Reliance,  with  then- 
oversight  bodies  for  implementation  (Joint  Directors  of  Laboratories;  the  area 
of  advanced  materials  was  added  during  the  implementation  phase  of  project 
Reliance). 
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•  A  new  management  and  planning  structure  will  implement  and  verify 
compliance  with  Reliance  agreements  through  the  Joint  Directors  of 
Laboratories  (JDL),  the  Armed  Services  Biomedical  Research,  Evaluation  and 
Management  (ASBREM)  Committee,  the  Training  and  Personnel  Systems 
Science  and  Technology  Evaluation  and  Management  (TAPSTEM) 
Committee,  and  the  Joint  Engineers. 

The  Tri-Service  S&T  Reliance  project  was  implemented  in  all  services 
by  November  25,  1991.  (Specific  early  accomplishments  of  Reliance  are 
reviewed  in  Attachment  B-1.) 

Tri-service  coordination  plans  have  been  developed  for  6.1  research 
programs  in  the  biological  and  medical  sciences,  cognitive  and  neural  sciences, 
and  the  environmental  sciences.  (A  list  of  all  tri-service  scientific  planning 
groups  is  provided  as  Attachment  B-2.) 


Conclusions 

While  it  is  too  early  to  draw  conclusions  about  Tri-Service  Reliance,  the 
subcommittee  believes  that  such  coordination  plans  show  promise  as  one 
mechanism  to  increase  the  effectiveness  and  productivity  of  research 
investment  dollars  across  DOD.  The  Report  of  the  1992  Defense  Science  Board 
Task  Force  on  Microelectronics  Research  Facilities^  for  example,  concludes  that 
Reliance  is  working  well  in  this  area.^  However,  the  committee’s  expert  view 
is  that  Reliance’s  achievements  to  date  have  been  variable.  Further 
development  of  Reliance  should  build  on  its  identified  successes;  and  its 
effectiveness  should  continue  to  be  assessed  and  ensured  through  appropriate 
cross-service  studies. 

The  Air  Force  participated  actively  in  project  Reliance.  It  should 
continue  to  study  and  support  new  initiatives  in  efficient^,  including 
cooperative  research  and  ways  to  reduce  duplication.  The  benefits  of  these 
efforts,  it  should  be  noted,  be  seen  more  over  the  long  than  the  short 
term. 

At  the  same  time,  the  Air  Force  should  not  give  tri-service  coordination 
inordinate  weight  in  its  planning  of  basic  research.  Satisfying  Air  Force  needs 
must  be  a  prior  goal. 


^  Report  of  the  1992  Defense  Science  Board  Task  Force  on  Microelectronics  Research 
Facilities,  report  to  the  Office  of  the  Under  Secretary  for  Acquisition,  June  1992. 
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OTHER  FEDERAL  AGENCIES 

In  carrying  out  its  comparative  analysis,  the  subcommittee  was  also 
briefed  by  representatives  of  non-DOD  agencies— the  Department  of  Energy 
(DOE),  National  Aeronautics  and  Space  Administration  (NASA),  and 
National  Science  Foundation  (NSF)-^n  selected  aspects  of  their  basic 
research  management,  which  are  discussed  first  below*  The  issues 
raised— primarily  concerning  the  use  of  external  peer  review  and  maintaining 
technics  managers’  skills— led  the  subcnmmittee  to  review  some  relevant 
operations  of  both  AFOSR  and  the  Air  Force  Scientific  Advisory  Board, 
which  are  discussed  subsequently.  Finally,  based  on  all  these  observations,  the 
subcommittee  offers  a  number  of  conclusions  toward  improving  the 
management  of  Air  Force  basic  research. 

Like  AFOSR,  DOE  and  NASA  are  mission  agencies,  and  their  basic 
research  programs  are  linked  to  their  specific  missions.  NSF  has  a  much 
broader,  general  mandate,  namely,  to  promote  and  advance  scientific  and 
engineering  progress  in  the  United  States. 


U.S.  Department  of  Energy 

In  DOE,  the  Director  of  the  Office  of  Energy  Research  (OER)  is 
Scientific  Advisor  to  the  Secretary  and  reports  directly  to  the  Secretary,  one 
indicator  of  the  prominence  that  the  agency  gives  to  basic  research. 

The  closest  analogue  to  AFOSR  within  DOE,  OER  funds  basic  research 
at  DOE  laboratories  and  universities  through  competitive  processes,  including 
external  peer  review.  Program  managers  make  funding  decisions  for  both  on- 
and  off-site  projects  based  on  the  recommendations  of  approximately  five 
external  reviewers.  The  reviewers  assess  proposed  projects  for  relevance  to 
DOE  and  for  scientific  merit,  and  submit  their  responses  by  mail.  For  review 
of  DOE  laboratory  proposals,  external  reviewers  are  selected  by  the 
laboratories  and  program  managers  together.  There  is  also  continual 
discussion  between  program  managers  and  the  laboratories  to  sharpen  the 
definition  of  project  areas,  both  before  proposal  submission  and  more 
generally. 

DOE’S  philosophy  is  to  make  long-term  commitments  to  all  its 
investigators  (with  fewer  new  starts),  especially  to  those  in  its  laboratories. 


®  This  comparative  study  focuses  on  selected  organizations  whose  missions  are  in  some 
important  way  similar  to  that  of  AFOSR.  However,  the  Air  Force  might  also  benefit  from  the 
practice  and  experience  of  other  agendes,  such  as  the  National  Institutes  of  Heedth  (NIH), 
which  supports  both  intra-  and  extramural  basic  research. 
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Off-site  projects  must  usually  be  renewed  every  three  years  by  submitting  new 
proposals.  On-site  laboratory  programs  must  submit  periodic  field  work 
proposals  (or  new  proposals  if  they  wish  to  change  research  significantly).  The 
lab  field  work  proposals  are  not  subjected  to  external  review.  All 
DOE-fimded  investigators  are  also  required  to  submit  annual  progress  reports 
and  professional  reprints. 

Another  DOE  technical  review  process,  for  ongoing  on-site  projects,  is 
that  used  by  units  within  OER’s  Office  of  Basic  Energy  Sciences  (BES).  In 
this  process,  external  reviewers  make  site  visits  to  laboratory  programs  roughly 
once  every  three  years  (visits  to  off-site  projects  are  more  irregular,  carried 
out  only  as  resources  allow).  Again,  this  review  is  conducted  by  program 
managers  and  reviewers  that  the  program  managers  and  laboratories  have 
jointly  identified.  Each  year,  some  part  of  every  lab’s  total  program  is 
reviewed  in  depth  in  this  way.  These  technical  site  reviews  are  advisory  only, 
and  their  main  object  is  to  work  in  cooperation  with  the  laboratories  to 
improve  the  quality  of  their  research. 

BES  research  managers  report  that  these  external  site  reviews  are  an 
important  contributor  to  the  quality  of  intramural  research.  In  part,  the 
reviews  are  needed  because  the  laboratories’  field  work  proposals  are  not 
externally  reviewed.  Beyond  this,  however,  this  form  of  review  allows 
obtaining  the  highest  quality  scientific  advice  (because  the  best  scientists 
anywhere  can  be  asked  to  participate),  as  well  as  some  independence  in 
review.  Moreover,  BES  representatives  observed,  these  in-depth,  external  site 
reviews  are  as  important  in  building  DOE  laboratories’  credibility  in  the 
scientific  community  as  they  are  in  building  quality.  BES  briefers  emphasized 
that  the  success  of  the  site  reviews  appears  to  depend  significantly  on  the 
nonadversarial,  advisory  relationship  between  the  reviewer  and  the  reviewed. 

DOE  program  manager  positions  in  the  basic  energy  sciences  are  stable, 
long-term  positions,  held  by  GS-15s  with  Ph.D.s.  The  briefers  identified  one 
weakness  of  this  management  pattern:  the  technical  skills  of  program 
managers  can  diminish  with  time  and  distance  from  active  research.  Program 
managers’  technical  skills  are  maintained  in  part  by  working  with  on-site 
researchers  and  by  attendance  at  one  or  two  high-level  professional  meetings 
annually.  However,  the  briefers  reported,  effort  is  required  on  the  program 
managers’  part  to  stay  abreast  of  developments  in  their  fields.  Some  DOE 
program  managers  suggest  that  additional  opportunities  (e.g.,  to  return 
periodically  to  hands-on  research)  would  better  help  them  maintain  strong 
professional  skills. 

DOE  laboratories  are  maintained  as  government-owned, 
contractor-operated  (GOCO)  facilities,  an  arrangement  that  some  experts 
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believe  offers  benefits  in  managing  research.’  The  DOE  briefers  noted  that, 
imder  this  system,  they  have  greater  flexibility  in  the  hiring  of  personnel.  They 
also  observed  that  DOE  contractor-operators  conduct  additional  technical 
quality  reviews  of  the  laboratories  they  operate. 

A  number  of  recent  reports  on  the  defense  technology  base  have 
reviewed  the  possible  advantages— as  well  as  disadvantages— of  GOCO  research 
management,  along  with  alternative  approaches  to  achieve  its  benefits  (see, 
e.g.,  the  earlier  cited  Defense  Science  Board,  1987;  Defense  Science  Board, 
1992;  Office  of  Technology  Assessment,  1989;  and  Report  to  the  Secretary  of 
Defense,  1991).  The  possible  use  of  GOCO-type  approaches  thus  appears  to 
remain  an  interesting  option  for  defensp  research  management. 


National  Aeronautics  and  Space  Administration 

NASA  is  distinguished  by  the  degree  to  which  its  management  of 
research  programs,  including  extramural  research  programs,  is  delegated  to 
the  NASA  centers.  NASA’s  unusual  form  of  management  was  established 
through  deliberate  policy  firom  the  agency’s  beginnings  (Office  of  Technology 
Assessment,  1989).  In  this  arrangement,  NASA  operates  through  a  network 
of  field  centers  (staffed  by  government  employees),  which  largely  define  the 
work  to  be  done,  select  contractors,  and  evaluate  the  work. 

The  overall  NASA  research  program  is  formulated  through 
headquarters’  issuance  of  "calls"  to  the  NASA  centers  and  announcements  of 
opportunity  (AOs)  to  industry  and  universities,  seeking  proposals  for  work  in 
specific  technology  areas.  These  areas,  which  are  specified  at  the  level  of 
technology  thrusts,  and  sometimes  at  the  level  of  specific  projects,  are 
developed  through  (1)  workshops  in  which  all  NASA  centers  participate;  (2) 
requests  for  identification  of  relevant  research  in  a  survey  of  operating  (or 
product)  divisions  of  NASA,  such  as  Space  Science,  Space  Transportation,  and 
Aeronautical  Technology;  and  (3)  the  discipline-oriented  reviews  of  the 
Aerospace  Research  and  Technology  committees  (ARTs),  made  up  of 
industry  and  university  experts.  The  projects  thus  established  form  the  basis 
of  NASA’s  budget  submissions,  and  also  of  the  agency’s  advocacy  before  the 
Executive  Branch  and  the  Congress. 

Funds  are  then  targeted  to  lead  NASA  centers  for  the  performance  or 
contracting  of  research  in  the  identified  technolog  areas.  Work  is  monitored 
by  headquarters’  discipline  leaders,  the  ARTs  groups,  and  on  an  atmual  basis 


’  NASA  has  also  used  such  an  arrangement  in  its  contract  with  the  California  Institute  of 
Technology  to  operate  the  Jet  Propulsion  Laboratory. 
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by  the  Space  and  Aeronautical  Systems  Technology  Advisory  committees  of 
NASA’s  Advisory  Council. 

NASA’s  Office  of  Aeronautics  and  Space  Technology,  for  example, 
which  briefed  the  subcommittee,  uses  sever^  forms  of  external  peer  review 
in  managing  a  mixed  program  of  research  and  technology  development.  These 
processes  may  provide  useful  models.  In  initial  program  planning  at  the  office 
level,  two  of  the  NASA  Council  advisory  committees  (the  Space  Systems  and 
Technology  Advisory  Committee  [SSTAC]  and  the  Aeronautical  Systems  and 
Technology  Advisory  Committee  [ASTAC]),  whose  members  are  research 
leaders  in  industry  and  universities,  meet  three  times  aimually  to  review  and 
provide  recommendations  on  general  program  area  thrusts.  Subcommittees 
look  at  specffic  discipline  areas  (e.g.,  photonics). 

Additionally,  the  ARTs  groups,  also  consisting  of  industry  and  university 
experts,  meet  separately  during  the  year  to  review  specific  projects.  (Like  the 
NASA  Council  advisory  committee  members,  ARTs  members  are  selected  by 
NASA  headquarters,  and  via  peer  recommendations.)  ARTs  groups  attempt 
to  examine  every  NASA  office  project,  both  ongoing  and  proposed.  They 
present  their  findings  annually  to  SSTAC  and  ASTAC.  These  findings  are 
advisory  only,  but  they  are  considered  important  and  taken  seriously. 

Office  budgetary  considerations  often  circumscribe  the  way  in  which  the 
various  NASA  advisory  groups’  findings  are  used.  However,  all  the  NASA 
external  review  processes,  at  both  program  and  project  levels,  are  considered 
valuable  by  the  community.  They  serve  as  a  respected  critique,  and  provide 
a  basis  for  advocacy  for  the  NASA  office  directors  and  NASA  Administrator. 


National  Science  Foundation 

The  broad  mission  of  the  NSF  is  mirrored  in  the  amount  of  its  research 
funding— $2.4  billion  in  1991,  as  compared  to  AFOSR’s  $170  million— and  also 
in  the  great  attention  it  has  given  to  developing  its  peer  review  procedures. 
Unlike  aU  other  agencies  examined  in  this  report  (with  the  exception  of 
ARO),  NSF-funded  research  consists  entirely  of  extramural  research 

The  high  level  of  competition  for  NSF  funding  has  been  one  factor 
leading  to  the  refinement  of  the  agency’s  review  procedures  over  the  years. 
Every  incoming  research  proposal  is  peer-reviewed,  in  contrast  to  the  practice 
in  other  agencies,  including  AFOSR,  where  proposals  are  first  screened  by 
program  management. 

As  in  the  other  agencies  here  reviewed,  NSFs  peer  review  of  proposals 
is  advisory  only,  with  program  officers  making  final  funding  recommendations, 
which  are  reviewed  at  the  directorate  level,  or  in  some  cases,  at  the  highest 
level  of  NSF  technical  oversight,  the  National  Science  Board,  However,  at 
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NSF,  as  compared  to  the  other  agencies,  greater  weight  is  given  to  the  views 
of  peer  reviewers  in  reaching  funding  decisions.  In  part,  this  difference  in 
practice  owes  to  NSFs  broad  (as  opposed  to  user-oriented)  mission 
orientation,  the  amount  of  research  it  funds,  and  the  high  level  of  competition 
for  its  research  funding.  These  characteristics  have  also  led  to  correspondingly 
high  levels  of  public  and  federal  attention  to  fairness  and  oversight. 

NSF  uses  four  main  criteria  in  evaluating  submitted  research  proposals: 
(1)  intrinsic  merit  of  the  proposed  research;  (2)  the  competence  of  the 
principal  investigator  (PI);  (3)  relevance  of  research,  for  applied  research;  and 
(4)  the  effect  of  the  proposed  research  on  the  science  and  engineering 
infrastructure  (human  resources,  specialized  facilities,  etc.). 

The  actual  mechanics  of  the  NSF  peer  review  process  varies  by  and 
within  NSF  directorate,  with  review  conducted  by  mail  or  in  panel,  or  both. 
Reviewers  are  experienced  Pis,  who  may  be  selected  by  program  managers, 
proposers,  or  other  reviewers.  (An  attempt  is  made  to  balance  panelists 
demographically.)  The  techniques  used  by  review  panels  also  vary  (e.g.,  they 
may  use  different  methods  of  review  and  ranking). 

However,  while  some  NSF  reviewing  techniques  are  allowed  to  vary,  all 
proposal  reviews  are  stiU  required  to  produce  certain  results.  Proposers  must 
receive  the  reviews  of  their  proposals,  and  the  anonymity  of  both  the  proposer 
and  the  reviewers  is  protected.  To  help  satisfy  the  last  requirement,  proposals 
are  technically  considered  NSF  intellectual  property.  Briefers  to  the 
subcommittee  noted  the  advantages  of  required  NSF  review  results,  including 
the  more  honest  responses  reviewers  are  likely  to  give  when  they  know  they 
will  remain  anonymous,  and  the  greater  fairness  assured  in  review,  with 
proposers’  being  provided  verbatim  copies  of  their  reviews  and  the  chance  to 
improve  and  resubmit  their  proposals. 

Despite  the  general  sophistication  and  soundness  of  NSF  review 
procedures,  their  development  has  also  led  to  a  cumbersome,  inefficient 
review  process  for  all  involved-proposers,  NSF  research  managers  and 
advisors,  and  the  reviewing  community.  One  main  conclusion  of  a  recent  NSF 
Merit  Review  Task  Force  was  that  "[NSF  review]  has  become  overburdened 
and  creativity  and  innovation  are  being  threatened."^”  The  task  force  also 
points  to  the  wide  agreement  in  this  view  in  the  scientific  and  engineering 
community. 

Among  other  problems  in  the  NSF  review  process,  both  the  briefers  and 
the  task  force  noted,  is  the  recent  great  rise  in  the  number  of  proposals 
submitted,  up  40  percent  in  the  last  decade.  At  the  same  time,  NSF  technical 
staff  have  remained  constant  in  number,  at  roughly  1,200  full-time  equivalent. 


Report  of  the  Merit  Review  Task  Force,  August  23, 1990,  National  Science  Foundation, 
Washington,  D.C. 
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Despite  various  improvements  in  NSF  review  efficiency  over  the  years,  the 
growing  numbers  of  submitted  proposals  have  resulted  in  heavy  administrative 
workloads  for  NSF  research  managers.  NSF  program  officers  now  usually 
manage  70  to  100  grants  (while  AFOSR  program  managers  typically  manage 
10  to  40). 

The  task  force  report  summarizes  the  many  problems  of  research 
management  that  result  from  NSFs  proposal  overload,  for  program  managers, 
and  also  for  Pis  and  reviewers; 

...Carefully  reading  (and  understanding)  the  proposal,  choosing 
reviewers,  documenting  specifics,  writing  recommendations  and 
review  summaries,  and  communicating  with  Pis,  can  receive  only 
minimum  attention.... 

Less  time  is  spent  on  maintaining  or  enhancing  professional 
expertise  in  the  field(s)  supported.  Minimal  time  is  available  for 
reading  (other  than  the  proposals  themselves),  for  site  visits  or  for 
attendance  at  specialized  conferences  and  symposia.  This  adversely 
affects  the  PC’s  ability  to  exercise  independent  judgment,  so  more 
reliance  must  be  placed  on  reviewers.  At  the  same  time,  it  restricts 
familiarity  with  the  emerging  reviewer  community,  so  established 
scientists  and  engineers  tend  to  be  used  more  often. 

Diminished  familiarity  with  the  proposed  research  (less  time 
available  for  each  proposal  in  addition  to  diminished  expertise) 
affects  not  only  the  decision  process  but  the  ability  to  communicate 
effectively  with  the  PI. 

NSF  does  have  specific  mechanisms  to  maintain  the  competence  of  its 
technical  managers,  for  example,  through  the  use  of  visiting  scientists  and 
engineers  as  program  managers,  and  through  independent  research  and 
development  agreements,  which  permit  visiting  scientists  to  remain  active  in 
research.  Such  activities  are  regarded  as  a  discharge  of  NSF  responsibilities 
and  are  clocked  as  regular  work  hours.” 

However,  the  NSF  task  force,  in  light  of  its  findings,  makes  a  number  of 
recommendations  to  lighten  program  managers’  administrative  workload  and 
improve  their  opportunities  to  maintain  technical  skills.  The  task  force 
recommends  that  program  managers  should  spend  about  20  percent  of  their 


”  NSF  Personnel  Manual,  Chapter  3  ("Employee  Development"),  March  21, 1989. 
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time  in  professional  development  activities,  and  permanent  staff  should  have 
regular  opportunities  for  research  leave. 

The  task  force  also  notes  the  challenges  of  NSF’s  extensive  use  of 
visiting  scientists  ("rotators")  as  research  managers  at  all  levels,  including  for 
the  management  of  peer  reviews.  Between  1983  and  1990,  such  visiting 
personnel  increased  from  22  to  40  percent.  While  their  presence  has  brought 
in  the  perspective  of  active  top  scientists  and  engineers,  and  valuable 
institutional  changes,  it  also  has  its  downside,  the  report  points  out:  "Visitors 
frequently  step  into  their  positions  with  limited  orientation  or  training....  The 
management  and  planning,  however,  needed  to  run  multimillion  dollar 
programs,  divisions  or  directorates  requires  a  corporate  infrastructure  that 
benefits  fi’om  e^qperience  as  much  as  firom  new  personnel  and  ideas  from  the 
outside." 

Both  briefers  to  the  subcommittee  and  the  task  force  reported  another 
result  of  the  increased  number  of  proposals  submitted  to  NSF:  while  average 
grant  size  has  not  changed  over  the  last  10  years,  in  constant  dollars  or 
duration,  the  costs  of  research  have  meanwhile  grown  significantly,  and  the 
average  current  NSF  grant  ($62,000)  fails  to  support  a  research  program 
adequately  (the  cost  of  education  and  training  of  a  single  graduate  student  is 
now  approaching  more  than  $40,000  annually).  Moreover,  according  to  the 
report,  the  growth  in  number  of  Pis  and  submitted  proposals,  coupled  with 
federal  funding  constraints,  leads  to  the  favoring  of  "safe"  proposals.  An 
earlier  report  also  identified  this  problem  and  pointed  out,  "to  require  that  the 
solutions...be  obvious  before  the  research  is  begun  discriminates  strongly 
against  innovative  work."^ 

One  NSF  program  director  who  briefed  the  subcommittee  had  previously 
served  as  a  program  manager  at  AFOSR.  Many  of  his  comments  were  in 
keeping  with  the  observations  above.  He  reported  that  AFOSR  does  a  better 
job  than  NSF  of  post-award  management  of  awards,  particularly  through  site 
visits,  because  of  the  large  number  of  grantees  in  NSF  programs.  AFOSR 
program  managers  can  take  more  chances  than  those  at  NSF  when  deciding 
on  awards.  One  notable  distinction  between  NSF  and  AFOSR  is  in  the 
amounts  of  their  financial  support.  In  recent  years,  AFOSR  has  generally 
provided  funds  adequate  to  do  a  proposed  research  project,  while  NSF  often 
partially  funds  research,  through  "grants-in-aid."  NSF  is  currently  thinking  to 
increase  its  grant  size,  because  of  the  difficulties  presented  by  current  award 
amounts.  This  briefer’s  summary  of  AFOSR’s  greatest  strengths  and 
weaknesses  from  a  comparative  point  of  view  are  provided  in  Table  B-1, 


White  House  Sdence  Council,  Panel  on  the  Health  of  U.S.  Colleges  and  Universities, 
"A  Renewed  Partnership,"  1986  (quoted  in  the  NSF  Merit  Review  Task  Force  Report). 
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TABLE  B-1  One  Program  Manager’s  View  of  the  Relative  Strengths  and 
Weaknesses  of  AFOSR  in  Research  Management 


Strengths 


Weaknesses 


Small,  efficient,  fiiendly 
organization 

Highly  qualified  technical  staff 

Respect  in  scientific  community 

Reasonable  workload;  time  for 
professional  development 

Travel  funds  adequate 

Varied  work;  interactions  with 
academic,  industrial,  DOD, 
and  other  government 
scientists  and  agencies 

Presubmission  contacts  and 
preproposals  reduce 
paperwork 

Funding  decisions  can  be  rapid 


Encumbered  by  DOD 
regulations  not  appropriate 
for  procuring  basic  research 

Contracting  process  too  long 
and  burdensome  for  PI 

Declining  purchasing  power 

Uncertain  budgets,  even  within 
fiscal  year 

Program  manager  grade/salary 
levels  a  source  of  discontent 

Other  restrictions  and 
deficiencies  due  to  U.S. 
government  salaries  and 
regulations 
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Clearly,  Air  Force  attempts  to  improve  its  basic  research  management 
should  benefit  from  the  lessons  offered  by  NSF  experience.  For  example,  the 
observation  that  heavy  workloads  and  the  long-term  stability  of  program 
manager  positions  can  diminish  technical  skills  is  also  in  keeping  with 
comments  made  by  Army,  as  well  as  DOE,  briefers.  Army  experts  pointed  out 
that  locating  the  Army  Research  Office  in  Research  Triangle  Park  has  been 
critical  in  maintaining  their  program  managers’  professional  skills  and  in 
increasing  the  credibility  of  their  research  management.  ARO  research 
managers  benefit  through  serving  as  adjunct  professors  in  the  Triangle 
universities. 

Maintaining  technical  competence  should  be  of  special  concern  to  the 
Air  Force,  with  the  reorganization  of  its  research  command  structure.  Care 
should  be  taken  to  avoid  overburdening  Air  Force  basic  research  managers 
with  new  procedural  and  process  demands.  Moreover,  in  a  1990  annual  report 
to  the  Air  Force  Acquisition  Executive  on  Air  Force  basic  research,  the  Air 
Force  Scientific  Advisory  Board  also  recommended  that  AFOSR  staff  should 
participate  in  a  "reverse  IPA  [Intergovernmental  Persoimel  Act  exchange]"  to 
remain  current. 


Air  Force  Office  of  Scientific  Research 

AFOSR  guidance  leaves  it  to  the  judgment  of  individual  program 
managers  whether  they  choose  to  have  basic  research  proposals  peer-reviewed 
(see  the  AFOSR  document  on  proposal  review,  in  Attachment  B-3,  and 
Attachment  A-1,  item  8).  Some  AFOSR  directorates  do  use  some  type  of 
formalized  external  proposal  review,  as  through  contracts  with  Universal 
Energy  Systems,  in  ffie  area  of  life  and  environmental  sciences,  and  with 
seversd  boards  of  the  National  Research  Coimcil,  in  the  areas  of  chemical  and 
atmospheric  sciences  and  high-energy-density  materials,  to  conduct  such 
advisory  reviews.  Proposals  are  first  screened  by  AFOSR  for  relevance  and 
quality  before  they  are  submitted  to  these  formal  external  reviews.  (One 
AFOSR  research  manager  involved  estimated  that  about  25  percent  of 
proposals  received  in  his  area  are  subjected  to  formal  external  review.)  In 
many  other  AFOSR  areas,  external  review  of  proposals  is  conducted  only  ad 
hoc. 

Those  AFOSR  personnel  consulted  who  have  been  involved  in  formal 
external  proposal  review  agreed  that  these  evaluations  are  valuable  in 
managing  their  own  programs  of  research.  They  reported  that  their  own 
external  proposal  reviews  are  not  expensive  (as  a  percentage  of  a  directorate’s 
budget),  and  that  such  reviews  help  to  identify  some  suboptimal  proposals  and 
also  some  proposals  whose  value  had  not  been  fully  recognized  by  AFOSR. 
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Also,  they  report,  the  use  of  external  peer  review  helps  build  the  credibility 
of  AFOSR’s  programs  in  their  own  scientific  communities. 

There  has  been  discussion  within  AFOSR  about  regularizing  its  external 
proposal  review;  and  members  of  the  subcommittee  are  highly  concerned  that 
AFOSR  research  should  be  able  to  withstand  some  form  of  external  peer 
review,  to  demonstrate  its  likely  scientific  and  engineering  value  for  the  Air 
Force  and  the  nation. 

At  the  same  time,  some  AFOSR  personnel  and  other  observers 
suggested  that  regularizing  AFOSR  external  proposal  review  could  raise 
problems.  Some  observers  pointed  out  that  professional  practices  vary  among 
scientific  fields  (especially  between  scientific  and  engineering  fields),  and 
these  "cultural  differences"  should  be  in  general  be  respected.  Relevance  to 
the  Air  Force  mission,  as  well  as  scientific  merit,  is  also  important  to  assess 
when  judging  the  value  of  Air  Force-funded  research,  even  in  the  case  of 
basic  research.  For  these  reasons,  some  with  experience  in  the  AFOSR 
environment  suggested  that  it  is  in  fact  appropriate  to  invest  the  program 
manager  with  significant  technical  decision-making  authority.  The  program 
manager  is  in  some  ways  uniquely  positioned  to  be  well  informed  about  both 
the  relevance  and  the  scientific  merit  of  Air  Force  basic  research  proposals. 
(The  few  who  argued  strongly  for  this  view  suggested  that  good  oversight  of 
AFOSR  technical  managers  is  therefore  a  critical  means  of  assuring  the 
quality  of  AFOSR  basic  research.) 

Others  pointed  out  that  conflicts  of  interest  can  characterize  external  as 
well  as  internal  review,  and  rigidly  formalized  external  review  can  lead  to 
overconservatism,  as  in  the  case  of  NSF,  which  may  be  especially  damaging 
for  basic  research.  Finally,  some  observed  that  regularized,  formal  external 
review  of  research  proposals  can  add  paperwork  burden,  again  as  in  the  case 
of  NSF,  and  related  delays  and  expense,  to  the  detriment  of  Air  Force 
research  management. 

It  is  unclear  whether  some  form  of  AFOSR  external  proposal  review 
should  be  regularized  or  conducted  AFOSR-wide,  though  Ais  subject  may 
well  deserve  further  examination.  A  well-designed  external  review  process  of 
some  kind  can  offer  strong  advantages,  including  significant  added  value  to 
the  nation  and  the  research  mission  of  the  Air  Force,  as  well  as  increased 
credibility  of  Air  Force  basic  research.  This  credibility  will  be  of  growing 
importance  in  maintaining  the  Air  Force’s  basic  research,  and  thus  its 
technologicsd  edge,  in  the  face  of  pressured  military  budgets.  At  the  same 
time,  inadequately  designed  external  review  could  add  new  burdens— notably, 
delays  and  additional  costs,  and  the  discouragement  of  innovation— to  the 
conduct  of  Air  Force  basic  research. 
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Air  Force  Scientific  Advisoiy  Board 

Yet  another  general  mechanism  for  technical  review  of  Air  Force  basic 
research  is  provided  through  Air  Force  Scientific  Advisory  Board  (SAB) 
review  of  planned  and  ongoing  6.1  programs.  The  Air  Force  Acquisition 
Executive  (SAF/AQ),  not  Air  Force  Materiel  Command,  assigns  SAB  the 
following  three  responsibilities: 

•  assessing  the  general  scope  and  direction  of  the  S&T  program; 

•  advising  on  technical  areas  (e.g.,  with  attention  to  emphases  and 
omissions);  and 

•  recommending  guidance  for  the  annual  AFAE  S&T  Executive 
Guidance  Memorandum.^^ 

In  the  area  of  Air  Force  basic  research,  SAB  has  fulfilled  these  roles,  for 
both  intra-  and  extramural  programs,  by  reviewing  the  Research  Technology 
Area  Plan  (TAP)  for  basic  research,  participating  in  the  spring  reviews  of  Air 
Force  Systems  Command  (to  which  AFOSR  until  recently  reported),  and 
visiting  AFOSR  in  the  fall. 

Tnfnrmation  provided  to  the  subcommittee  indicates  that  these  SAB 
advisory  procedures  are  considered  successful  and  valuable.  However,  some 
SAB  representatives  and  members  of  the  subcommittee  are  concerned  that 
SAB  processes,  as  they  stand,  may  fail  to  provide  sufficient  external,  in-depth 
technical  review  of  Air  Force  basic  research  programs.  SAB  reviews  are  for 
the  most  part  general,  assessing  entire  research  program  areas,  rather  than 
specific  projects,  with  only  occasional  in-depth  examir:,-iion  of  particular  areas 
(e.g.,  when  the  specific  technical  expertise  of  an  SAB  review  panelist  an«-  the 
subject  of  an  AFOSR  project  are  a  good  match). 

It  may  well  be  that  more  in-depth,  external  review  of  ongoing  specific 
projects  is  the  best  means  for  the  Air  Force  to  achieve  greater  technical 
perspective  and  oversight  on  the  value  of  its  basic  research.  As  observed  in 
the  discussion  on  DOE,  external  project  review  allows  research  in  progress  to 
be  improved,  encourages  follow-on  to  specific  promising  research  thrusts,  and 
helps  identify  research  emphases  that  have  become  outmoded.  (While 
discussions  with  DOD  representatives  did  not  lead  the  subcommittee  to 
examine  Army  or  Navy  review  processes  in  any  depth,  it  should  be  noted  that 
Navy  briefers  did  note  the  importance  of  the  Navy’s  external  review  processes 
for  basic  research;  and  various  experts  have  also  observed  that  the  Army 
Research  Office’s  responsibility  for  extramural  projects  alone  allows  ARO  to 


^  U.S.  Air  Force  Scientific  Advisory  Board  Task  on  Science  and  Technology  Broad 
Program  Appraisal,  September  17,  1990. 
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serve  as  an  "honest  broker"  in  assessing  the  Army’s  intramural  research 
programs.) 


Conclusions 

Information  provided  by  non-DOD  representatives— from  DOE,  NASA, 
and  NSF— supports  the  following  conclusions  concerning  the  management  of 
Air  Force  basic  research: 

1.  The  Air  Force  should  strongly  consider  incorporating  greater 
external,  in-depth,  technical  review  in  managing  its  research  programs,  both 
intra-  and  extramural.  Mechanisms  now  in  place  to  provide  such  review 
appear  to  be  working  well,  so  far  as  they  go.  But  there  are  a  number  of 
indications  that  these  mechanisms  may  not  be  enough.  Research  today  is 
funded  and  conducted  much  more  widely  throughout  the  world  than  it  was  20 
years  ago,  and  it  is  increasingly  easy  for  research  programs  to  be  isolated  from 
relevant  research  developments.  >^^e  the  subcommittee  appreciates  the  real 
uniqueness  and  value  of  the  technical  decision-making  role  of  the  AFOSR 
program  manager,  it  is  concerned  that  current  review  of  AFOSR  projects  may 
rest  too  greatly  on  the  views  of  too  few  individuals— however  skilled  and  well 
informed  they  may  be  as  individuals.  (We  also  note  that,  according  to  the 
information  provided,  some  elements  of  the  AFOSR  program  are  given  very 
Uttle  or  no  external  review  of  any  sort.) 

2.  Moreover,  other  federal  agencies,  including  the  mission  agencies 
DOE  and  NASA,  are  able  to  achieve  greater  external,  in-depth  technical 
review,  and  report  that  such  review  significantly  improves  the  quality  and 
credibility  of  their  research  programs.  Both  these  achievements  will  be 
increasingly  critical,  if  the  Air  Force  is  to  satisfy  its  research  mission  in  a  more 
highly  pressured  budgetary  environment. 

3.  In  sum,  greater  external  peer  review  can  help  greatly  in  determining 
that  Air  Force  basic  research  does  not  duplicate  research  done  by  others,  that 
it  stays  informed  of  and  capitalizes  best  on  outside  research-related 
developments,  and  that  it  is  of  the  highest  quahty  generally.  At  the  same  time. 
Air  Force  review  mechanisms  must  be  designed  to  ensure  that  Air  Force  basic 
research  is  best  responsive  to  Air  Force  needs,  including  the  needs  to  satisfy 
mission  requirements,  contain  costs,  and  avoid  new  bureaucratic  burdens  or 
delays. 

4.  The  subcommittee  suggests  that  further  attention  be  given  to 
achieving  greater  external  technical  review  of  Air  Force  research,  taking  into 
account  the  views  and  operations  of  informed  communities,  such  as  AFOSR, 
Air  Force  laboratories,  AFOSR-funded  researchers  (representing  all  research 
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environments),  SAB  representatives,  and  relevant  elements  of  the  larger 
research  community.  This  discussion  should  include  a  more  detailed 
evaluation,  as  appropriate,  of  particular  models  used  by  others,  such  as  DOE, 
NASA,  and  NSF.  The  subject  of  external  technical  review  is  an  important  and 
complex  one  for  all  concerned,  as  well  as  for  the  Air  Force  as  a  whole;  it 
deserves  to  be  addressed  in  corresponding  depth. 

5.  In  view  of  observations  concerning  DOE,  and  especially  NSF, 
program  management,  AFOSR  program  managers  must  be  given  the 
workload,  span  of  control,  and  professional  opportunities  that  will  allow  them 
to  continuaJly  renew  their  teclmical  skills  and  standing  in  their  disciplines. 
These  needs  are  proportionally  greater  in  the  degree  that  technical  decision¬ 
making  authority  is  invested  in  AFOSR  research  managers.  In  changes 
following  on  its  recent  reorganization,  the  Air  Force  should  particularly 
ensure  that  program  managers  in  basic  research  are  not  overburdened  with 
new  procedural  and  process  demands. 

6.  Because  of  the  potential,  but  uncertain,  net  advantages  of  GOCO 
management,  the  Air  Force  should  keep  in  mind  the  further  exploration  of 
GOCO-type  arrangements  for  the  management  of  its  basic  research. 
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ATTACHMENT  B-l: 

EARLY  ACCOMPLISHMENTS  OF  THE  TRI-SERVICE  SCIENCE  AND 
TECHNOLOGY  RELIANCE  PROJECT 

Excerpt  from  the 

"White  Paper  on  Tri-Service  Reliance  in 
Science  and  Technology" 

Joint  Directors  of  the  Laboratories 
Department  of  the  Army 
Department  of  the  Navy 
Department  of  the  Air  Force 

January  1992 


•  Collocation  of  all  Training  Devices  and  Aircrew  Training  S&T  in 
Orlando,  Florida.  This  increase  in  Tri-Service  Reliance  created  a 
Tri-Service  Center  of  Excellence  and  eliminates  multiple  sites 
performing  similar  work, 

•  Collocation  of  all  S&T  activity  in  Survivability  and  Protective 
Structures  at  a  single  site,  the  Amy  Waterways  Experiment  Station 
(WES),  Vicksburg,  Mississippi.  This  increase  in  Tri-Service  Reliance 
eliminated  redundant  capabilities  and  permitted  the  reinvestment  of 
resources  to  strengthen  other  important  S&T  areas. 

•  Collocation  of  all  conventional  Guns  S&T  within  the  Amy  at  the 
Amament  Research,  Development  and  Engineering  Center 
(ARDEC),  Picatinny  Arsenal,  Dover,  New  Jersey, 

•  Collocation  of  all  Fuels  and  Lubes  S&T  to  Wright  Laboratory  at 
Wright  Patterson  Air  Force  Base,  Ohio.  This  increase  in  Tri-Service 
Reliance  involves  the  Amy  collocating  its  Fuels  and  Lubes  program 
from  Belvoir  Research,  Development  and  Engineering  Center 
(BRDEC)  in  order  to  strengthen  the  Tri-Service  Program  while  still 
meeting  the  enduring  mission-essential  requirements  of  the  Services. 

•  Collocation  of  Amy  Health  Effects  research  with  the  Air  Force  and 
Navy  Toxicolo©r  Programs  to  Armstrong  Laboratory  at 
Wright-Patterson  Air  Force  Base,  Ohio.  This  increase  in  Tri-Service 
Reliance  consolidates  major  portions  of  medical  S&T  through 
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collocation  at  single  sites  and  creates  Tri-Service  Centers  of 
Excellence. 

•  Collocation  of  in-house  S&T  work  addressing  Space-based 
Wide-Area  Surveillance  Radar  at  the  Air  Force  Rome  Laboratory, 
Rome,  New  York. 

•  Collocation  of  in-house  S&T  work  addressing  Space-based  Infra-red 
Sensors  for  Wide-Area  Surveillance  at  the  Naval  Research 
Laboratory,  Washington,  D.C. 

•  Collocation  of  all  directed  Energy  Bioeffects  S&T  of  the  Army  and 
Navy  to  Armstrong  Laboratory  at  Brooks  Air  Force  Base,  Texas. 

•  Collocation  of  all  Biodynamics  S&T  of  the  Army  and  Navy  to 
Armstrong  Laboratory  at  Wright-Patterson  Air  Force  Base,  Ohio. 

•  Collocation  of  all  Army  Combat  Dentistry  S&T  with  the  Navy  in 
Great  Lakes,  Illinois. 

•  Collocation  of  Army,  Navy,  and  Air  Force  6.1  Foreign  Field  offices 
and  the  development  of  coordinated  science  monitoring  programs. 

•  Establishing  JDL  Centers  of  Excellence  in  Artificial  Intelligence. 

•  Conducting  inter-Service  competition  for  DARPA  Supercomputer 
hardware. 

•  Providing  an  effective  Service  focal  point  for  developing  the  DoD 
Software  Technology  Plan. 

•  Providing  effective  Tri-Service  coordination  with  NASA,  the  Federal 
Aviation  Authority,  and  the  National  Security  Agen<^,  and 

•  Conducting  the  1991  OSD  Science  and  Technology  Reviews  using 
the  Reliance  infrastructure. 
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ATTACHMENT  B-2; 

FISCAL  YEAR  1992  TRI-SERVICE 
SCIENTIFIC  PLANNING  GROUPS  (SPGS) 


PHYSICS 

Chairman:  Dr.  B.D.  Guenther,  ARO 
Dr.  Sidney  L.  Ossakow,  ONR 
Dr.  Horst  R.  Wittmann,  AFOSR 

CHEMISTRY 

Chairman:  Dr.  Ronald  A.  DeMarco, 
ONR 

Dr.  Donald  L.  BaU,  AFOSR 
Dr.  Robert  G.  Ghirardelli,  ARO 

MATHEMATICAL  SCIENCES 
Chairman:  Dr.  Charles  J.  Holland, 
AFOSR 

Dr.  Neil  Gerr,  ONR 
Dr.  Jagdish  Chandra,  ARO 

COMPUTER  SCIENCE 
Chairman:  Dr.  Jagdish  Chandra,  ARO 
Dr.  Charles  J.  Holland,  AFOSR 
Dr.  A.M.  van  Tilborg,  ONR 

ELECTRONICS 

Chairman:  Dr.  G.M.  Borsuk,  ONR 

Dr.  J.W.  Mink,  ARO 

Dr.  Horst  R.  Wittmann,  AFOSR 

MATERIALS  SCIENCE 
Chairman:  Dr.  Allan  Rosenstein,  AFOSR 
Dr.  Robert  Pohanka,  ONR 
Dr.  Andrew  Crowson,  ARO 

Chair  rotates  in  foUowing  years: 
ARO/ONR/AFOSR 


ATMOSPHERIC  AND 

SPACE  SCIENCES 

Chairman:  Dr.  Walter  A.  Flood,  ARO 

LtCol  James  G.  Stobie,  AFOSR 

Dr.  Robert  F.  Abbey,  Jr.,  ONR 

BIOLOGICAL  AND 
MEDICAL  SCIENCES 
Chairman:  Dr.  Robert  W.  Newbrngh, 
ONR 

Dr.  William  O.  Berry,  AFOSR 
Dr.  Shirley  R.  Tove,  ARO 

COGNITIVE  AND 

NEURAL  SCIENCES 

Chairman:  Dr.  William  O.  Berry,  AFOSR 

Dr.  W.S.  Vaughn,  ONR 

Dr.  Robert  S.  Campbell,  ARO 

MECHANICS 

Chairman:  Dr.  Robert  E.  Singleton,  ARO 
Dr.  Jim  C.I.  Chang,  AFOSR 
Dr.  Spiro  Lekoudis,  ONR 

OCEAN  GEOPHYSICS  AND 
TERRESTRIAL  SCIENCES 
CoChairman:  Dr.  Steven  Mock,  ARO 
CoChairman:  (TBD),  ONR 

ENVIRONMENTAL  QUALITY 
SCIENCES 

Chairman:  Dr.  William  O.  Berry,  AFOSR 
Dr.  Robert  W.  Newburgh,  ONR 
Dr.  Shirley  R.  Tove,  ARO 
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ATTACHMENT  8-3: 

PROPOSAL  REVIEW  IN  THE  AIR  FORCE 
OFFICE  OF  SCIENTIFIC  RESEARCH 

AFOSR’s  Process  of  Reviewing  Proposals  in  the  Air  Force: 

The  Air  Force  Office  of  Scientific  Research  (AFOSR)  invites  proposals  with  a  Broad  Agency 
Announcement  (BAA)  for  basic  research  in  support  of  the  Air  Force  Defense  Research 
Sciences  Program.  The  general  areas  of  interest  are: 

Aerospace  Sciences:  structural  dynamics,  mechanics  of  materials,  structural  mechanics, 
particulate  mechanics,  external  and  internal  aerodynamics,  turbulence  structiue  and 
control,  unsteady  and  separated  flows,  air-breathing  combustion,  rocket  and  space 
propulsions,  and  diagnostics  in  reacting  media. 

Chemistry  and  Materials  Science:  electrochemistry,  inorganic  materials,  polymer 
chemistry,  molecular  dynamics,  chemical  synthesis  and  reactivity,  theoretical  chemistry, 
metallic  structural  materials,  and  ceramic  and  nonmetallic  structural  materials. 

Physics  and  Electronics:  atomic  and  molecular  physics,  optical  and  plasma  physics, 
electronic  and  photonic  materials  and  devices,  superconductivity,  and  optical  signal 
processing. 

Life  and  Environmental  Sciences:  regulation  of  neuronal  responsiveness,  neural  bases  of 
behavior,  computational  neuroscience,  chronobiology,  cognitive  processes,  sensory  and 
perceptual  science,  spatial  orientation,  bioenvironmental  science,  toxic  mechanisms, 
ionospheric,  atmospheric  and  space  environmental  sciences,  meteorological  research,  and 
optical  and  infrared  environment  research. 

Mathematical  and  Computer  Sciences:  dynamics  and  control,  physical  mathematics  and 
applied  analysis,  computational  mathematics,  optimization  and  discrete  mathematics, 
signal  processing,  probability  and  statistics,  software  and  systems,  artificial  intelligence, 
neural  computation  systems,  and  electromagnetics. 

The  primary  evaluation  factors  follow: 

1.  The  scientific  and  technical  merits  of  the  proposed  research. 

2.  The  potential  contributions  of  the  proposed  research  to  the  mission  of  the  Air  Force. 

3.  Availability  of  funds. 

Other  evaluation  criteria  include: 

1.  The  likelihood  of  the  proposed  eflfort  to  develop  new  research  capabilities  and  to  broaden 
the  research  base  in  support  of  national  defense. 
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2.  The  offeror’s  capabilities,  related  experience,  facilities  or  techniques  or  unique 
combinations  of  these  factors  that  are  integral  to  achieving  the  objectives. 

3.  The  qualifications,  capabilities,  experience,  and  past  research  accomplishments  of  the 
proposed  Principal  Investigator,  team  leader  or  key  personnel  who  are  critical  to  the  DOD 
mission. 

4.  Realism  and  reasonableness  of  proposed  cost. 

5.  Offeror’s  record  of  past  research  accomplishments. 

Policy  and  Process  on  Approval  Authority  of  Sponsorship  of  Research 

Sponsorship  will  only  be  approved  for  research  proposals  aimed  at  both  research  excellence  and 
Air  Force  relevance. 

Research  proposals  which  enhance  educational  opportunities  and  strengthen  the  nation’s  base 
of  mechamcally  trained  personnel  (such  as  through  graduate  assistantships  and  support  of 
graduate  student  research  programs)  should  be  evduated  more  favorably  than  research 
proposals  of  equal  technical  merit  which  do  not  provide  this  component. 


Upon  the  completion  of  a  program  manager’s  favorable  review,  the  program  manager  makes 
a  recommendation  (including  funding  recommendation)  to  the  Director  within  his/her 
directorate.  The  AFOSR  Scientific  Directors  have  the  authority  to  approve  the  use  of  ftmds 
for  all  US  domestic  actions  up  to  $1  million  in  the  aggregate.  Actions  exceeding  this  limit  as 
well  as  all  sponsorship  of  foreign  research  require  approval  by  the  Director,  AFOSR. 

All  sponsorship  must  conform  to  the  budget  released  by  the  Plans  and  Programs  Division, 
XOP,  and  is  subject  to  the  availability  of  funds.  Thus,  sponsorship  (release  of  funds),  requires 
approval  by  the  Chief,  Resource  Management  Division. 

The  Director  of  Contracts,  PK,  is  authorized  to  approve  all  grants  and  contracts  in  conformance 
with  applicable  AFSC  FAR  Supplements  (currently  up  to  $2  million  for  grants,  $3  million  for 
contracts).  This  authority  may  be  redelegated  in  accordance  with  the  AFSC  FAR  Supplements. 

The  following  chard.  Management  of  Air  Force  Basic  Research,  depicts  AFOSR’s  direct 
process,  coordination  and  policy  of  how  basic  research  funds  are  spent,  and  the  funding  flow 
process. 


Appendix  C: 

Quality  of  AFOSR-Sponsored  Research 


C.  Kumar  N.  Patel,  Chairman 
Gilbert  F.  Decker 
Daniel  P,  Schrage 
Charles  Zraket 

One  charge  to  the  Committee  on  Air  Force  Research  Management  was 
to  evaluate  the  quality  of  Air  Force  6.1  (basic  research)  programs,  that  is, 
programs  sponsored  by  the  Air  Force  Office  of  Scientific  Research  (AFOSR). 
The  report  of  this  subcommittee  specifically  addresses  this  charge. 

The  subcommittee  considered  in  particular  the  relative  quality  of  AFOSR 
intra-  and  extrammal  research.  It  also  touched  on  another  study  charge  in  the 
course  of  its  work:  assessing  the  means  and  extent  of  technology  transition  by 
performers  of  Air  Force  6.1  research.  The  practical  value  of  research— as  seen 
in  technology  transition— is  one  of  the  most  important  ultimate  measures  of 
research  quality. 

Clearly,  high-quality  basic  research  has  a  significant  impact  on  Air  Force 
technology  and  mission  accomplishment.  At  the  same  time,  ensuring 
high-quality  research  requires  the  use  of  appropriate  quality  metrics,  which 
are  difficult  to  specify  for  basic  research.  Good  basic  research  is  often  driven 
more  by  scientific  than  practical,  goal-oriented  motivations;  and  its  full  impact 
may  not  be  evident  for  many  years. 

As  a  first  step,  the  subcommittee  therefore  identified  a  variety  of 
measures  that  can  be  useful,  if  imperfect,  indicators  of  basic  research  quality 
(Attachment  C-1).  They  concern  the  quality  of  research  personnel,  the  quality 
of  research  results,  technology  transition,  contribution  to  the  knowledge 
infirastructure,  and  assurance  of  long-term  scientific  manpower  needs.  The 
identified  measures  include  those  used  by  universities,  private  research 
laboratories,  and  peer  review  and  accrediting  committees  in  their  assessments 
of  basic  research  quality.  These  measures  also  include  those  that  the 
subcommittee  identified  with  AFOSR’s  particular  needs  in  mind. 

After  outlining  AFOSR’s  current  review  processes  below,  the 
subcommittee  evaluates  these  processes  in  light  of  measurement  criteria 
selected  from  the  subcommittee’s  identified  list.  Caution  is  urged  in  the  use 
of  the  suggested  criteria,  because  none  of  them  alone  provides  a  full  picture. 
At  the  same  time,  each  has  some  validity,  and  a  variety  of  these  criteria  taken 
together  can  offer  a  good  starting  point  for  better  measurement,  and  thus 
management,  of  Air  Force  basic  research. 
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CURRENT  AFOSR  REVIEW  PROCESSES 

AFOSR  fully  recognizes  that  developing  and  implementing  an  optimal 
review  process  for  Air  Force  basic  research  is  a  multidimensional,  complex 
issue.  Factors  contributing  to  this  complexity  include  the  following: 

•  the  wide  range  of  disciplines  in  which  the  Air  Force  sponsors  basic 
research; 

•  the  large  number  of  Air  Force  6.1  research  projects,  or  work  units, 
imderway  at  any  given  time  (roughly  1,500  work  units^);  and 

•  the  large  number  of  institutions,  geographically  widely  dispersed,  at 
which  Air  Force  basic  research  is  conducted. 

AFOSR  does  cany  out  its  own  audit  of  the  quality  and  effectiveness  of 
the  research  it  supports,  by  means  of  review  processes  that  have  been  refined 
and  improved  over  the  last  three  years: 

•  New  review  responsibilities  have  been  defined  for  AFOSR’s  program 
managers  and  the  scientific  directors  to  whom  they  report.  Quality  standards 
and  the  review  responsibilities  of  program  managers  and  scientific  directors 
have  been  clarified. 

•  "Relevancy"  reviews  are  conducted  at  each  of  the  Air  Force 
laboratories  annually  during  the  fall;  in  these  reviews,  the  laboratories 
evaluate  the  AFOSR  basic  research  programs  in  general,  with  emphasis  on 
relevance  to  Air  Force  needs. 

•  Quality  reviews  are  also  held  at  these  fall  meetings,  with  quality 
review  forms  being  completed  by  invited  members  of  the  Air  Force  Scientific 
Advisory  Board  and  the  AFOSR  scientific  directors. 

•  The  relevancy  review  inputs  and  the  quality  review  inputs  are 
compiled  by  AFOSR  into  two  overall  numerical  ratings  for  each  project,  one 
for  excellence  (or  quality),  the  second  for  relevance. 

•  In  a  spring  review  also  held  by  AFOSR,  general  progress  of  ongoing 
programs  is  assessed,  new  initiatives  are  considered  (this  is  advertised  as  the 
point  where  innovative  ideas  can  be  addressed),  and  future  plans  for  ongoing 
programs  are  reviewed. 

To  better  assess  the  effectiveness  of  AFOSR’s  review  processes  in  action, 
several  members  of  the  study  committee  attended  three  of  the  four  fall 


^  Headquarters,  Air  Force  Materiel  Command,  FY  93  Research  Technology  Area  Plan, 
DCS/Sdcnce  and  Technology,  Wright  Patterson  Air  Force  Base,  Ohio. 
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reviews  held  at  the  Air  Force  laboratories.  The  following  observations  were 
made: 


•  Because  of  the  many  work  units  covered,  only  research  abstracts 
were  presented  at  the  meeting;  there  was  effectively  no  in-depth  review  of  the 
research. 

•  No  researchers  from  AFOSR’s  extramural  programs  were  present  at 
the  reviews. 

•  Attendance  by  external  "peer"  reviewers,  including  members  of  the 
Air  Force  Scientific  Advisory  Board,  was  limited. 


Findings 

Based  on  the  information  it  was  provided  through  the  briefings  and  site 
visits,  the  subcommittee  reached  the  following  findings: 

1.  Significant  improvements  have  been  made  in  the  last  few  years  in 
AFOSR’s  review  processes,  especially  in  evaluating  the  relevancy  of  basic 
research  to  Air  Force  needs. 

2.  The  fall  reviews  are  fundamentally  internal  reviews,  and,  while  they 
provide  useful  indicators  of  relevancy,  they  provide  questionable  indicators  of 
quality,  and  little  or  no  in-depth  review  of  technical  subject  matter. 

3.  The  review  of  research  quality  is  very  highly  decentralized,  depending 
almost  exclusively  on  the  capabilities  and  initiative  of  the  responsible  program 
manager. 

4.  The  failure  to  apply  any  quantitative,  and  also  some  qualitative, 
measures  of  research  quality  (as  described  below)  makes  the  evaluation  of 
AFOSR-sponsored  research  very  difficult. 

5.  Little  dependence  is  placed,  either  formally  or  informally,  on  the 
views  of  outside  review  groups. 

6.  Some  information  is  available  on  the  quality  of  the  basic  research 
program,  as  measured  by  the  current  review  process,  but  it  has  not  been 
integrated  into  overall  statistics  nor  independently  validated. 

7.  There  is  very  little  overall  strategic  planning  for  AFOSR’s  basic 
research  program.  V^ile  many  program  managers  appear  to  do  a  credible 
job  of  managing  within  their  own  domains,  overall  goal-setting  in  light  of 
long-term  Air  Force  needs  appears  not  to  receive  adequate  attention. 


78 


MANAGING  AIR  FORCE  BASIC  RESEARCH 


CRITERIA  TO  EVALUATE  RESEARCH  QUALITY 

Having  examined  current  AFOSR  review  processes,  the  subcommittee 
used  some  of  the  measurement  criteria  it  had  identified  (Attachment  C-1)  to 
benchmark  these  processes.  The  section  below  discusses  these  selected  criteria 
and  presents  related  observations  and  findings  about  AFOSR  review. 


Personnel 

In  mature  fields  of  science,  interesting  problems  become  increasingly 
complex,  often  requiring  team  efforts  to  arrive  at  solutions.  Even  so,  first-rate 
research  teams  are  generally  composed  of  first-rate  individual  researchers.  A 
number  of  objective  measures  can  be  used  to  evaluate  the  performance  of 
both  the  individual  researcher  and  the  research  team. 


Obi ‘active  Measures 

Objective  quality  metrics  for  individual  research  personnel  include  the 
following: 

•  fellowships  in  professional  societies; 

•  memberships  in  national  academies;  and 

•  other  forms  of  professional  recognition,  such  as  society  prizes  and 
awards. 

These  criteria  are  characteristic  of  the  highest  caliber  researchers,  reflecting 
peer  evaluation  of  the  sum  total  of  an  individual’s  contributions  to  date. 
Researchers  in  the  best  university  research  groups  tend  to  rank  very  high 
according  to  these  criteria. 

The  following  group  measures  can  also  be  useful  in  assessing  the  quality 
of  research  personnel: 

•  breakdown  by  advanced  degrees  and  scientific  and  engineering 
disciplines;  and 

•  average  professional  age  (and  age  distribution  by  fields  and  degrees). 

While  past  performance  may  not  be  a  perfect  indicator  of  future 
accomplishments,  the  above  measures  do  provide  some  collective  insight  into 
the  quality  of  people  doing  research. 
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The  subcommittee  was  unable  to  obtain  information  regarding  any  of  the 
above  measures  for  AFOSR-supported  personnel.  It  appears  that  such  data 
are  not  routinely  compiled  for  any  AFOSR-supported  research  personnel, 
whether  imiversity  faculty,  industry  researchers,  or  Air  Force  laboratory 
personnel.  This  is  unfortunate.  The  absence  of  such  data  for  Air  Force 
laboratory  persoimel  in  particular  makes  it  especially  difficult  to  draw  any 
significant  conclusions  about  the  overall  qudity  of  researchers  AFOSR 
supports. 

In  this  context,  it  is  important  to  note  AFOSR’s  use  of  the  Star  Team 
Award,  which  recognizes  excellence  in  basic  research  in  the  Air  Force 
laboratories.  These  awards  are  based  in  part  on  some  of  the  quality  criteria 
identified  here  and  below.  They  appear  to  be  a  well-designed,  valuable  tool 
for  rewarding  and  promoting  excellence  in  research.  At  the  same  time,  they 
are  not  designed  to  serve  as  general  measures  of  the  quality  of  all  Air  Force 
basic  researchers  or  of  all  Air  Force  basic  research. 


Subjective  Observations 

AFOSR  did  provide  the  information  it  had  readily  available  on  the 
research  persoimel  it  supports.  This  information  consisted  of  a  nearly 
complete  list  of  current  AFOSR  research  projects  and  grants,  by  subject  title, 
name  of  the  principal  investigator  (PI),  and  the  PFs  institution.  Based  on 
these  data,  the  subcommittee  was  able  to  draw  some  general  qualitative 
inferences  about  the  quality  of  research  personnel  to  whom  AFOSR  awards 
grants  and  projects. 

The  grants  are  distributed  over  a  very  large  number  of  academic 
institutions.  By  itself,  this  is  not  an  indicator  of  personnel  quality,  but  an 
indicator  of  program  diversity.  However,  the  number  of  grants  to  better 
known,  highly  rated  research  institutions  appears  proportionally  greater  than 
the  number  of  grants  to  less  well  known  institutions.  Pis  at  these  highly  rated 
institutions  are  more  likely  to  be  top-quality  researchers,  given  their 
institutions’  high  standards,  and  thus  high  standards  are  likely  being  used  by 
AFOSR  program  managers  in  their  grant  selection.  A  review  of  the  names  of 
university  researchers  supported  by  AFOSR,  in  fields  familiar  to  the 
subcommittee,  indicates  that  well-known  individuals  are  often  carrying  out  the 
research  (though  in  some  cases  the  researchers  are  past  their  most  productive 
period  of  research).  It  is  more  difficult,  of  course,  to  assess  how  many  of  the 
researchers  unfamiliar  to  the  subcommittee  are  those  who  are  up-and-coming 
in  their  fields  but  not  yet  well  known. 

Discussions  with  the  Director  of  AFOSR  and  several  research  program 
managers  suggest  that,  in  AFOSR’s  consideration  of  grant  proposals  and 
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review  of  research  content  and  progress,  it  is  generally  left  to  the  individual 
program  manager  to  assess  the  quality  of  research  personnel.  Some  program 
managers  reportedly  tiy  to  apply  more  rigorous  methods  than  others  in 
assessing  researcher  qu^ity.  Again,  no  systematic  database  is  maintained  on 
the  objective  quality  metrics  for  research  personnel  that  were  identified  above. 


Finding 

The  lack  of  an  organized  database  on  objective  quality  metrics  is  a 
serious  deficiency  in  attempting  to  evaluate  the  overall  quality  of  researchers 
to  whom  AFOSR  awards  grants  and  projects.  Without  such  a  database,  the 
assurance  of  quality  standards  is  left  to  the  highly  subjective  judgment  of 
individual  program  managers.  The  sound  evaluation  of  research  personnel 
requires  a  more  objective  basis. 


Research  Results 

One  critical  outcome  of  research  is  new  knowledge,  which  provides  the 
foundation  for  new  technologies.  An  important  mechanism  for  disseminating 
and  validating  new  knowledge  is  publication  in  refereed  journals.  The  full 
impact  of  a  publication— its  contribution  to  the  knowledge  base— is  difficult  to 
assess  without  an  in-depth  xmderstanding  of  the  relevant  field.  However,  a 
helpful  and  widely  used,  if  imperfect,  measure  of  a  publication’s  impact  is  the 
frequency  with  which  it  is  cited  by  peers. 

Unfortunately,  such  publication  and  citation  data  are  not  kept  by  AFOSR 
on  the  work  it  supports,  even  as  a  rolling  data  bank  that  may  look  back  three 
to  five  years.  The  quality  of  AFOSR  research  results  is  also  assessed  largely 
through  the  judgments  of  individual  AFOSR  program  managers.  Again,  for 
the  well-known  researchers  supported  by  AFOSR,  the  subcommittee  can  be 
reasonably  sure  that  their  publications  have  value,  but  only  in  these  cases  and 
in  familiar  fields.  Statistical  data  on  publications  and  citations  for  AFOSR- 
sponsored  work  would  provide  better  quality  measures  of  research  results. 
These  data  would  make  possible,  for  example,  a  more  rationally  based  assess¬ 
ment  of  the  relative  contributions  of  intra-  and  extramurally  conducted 
research. 

Another  useful  measure  of  the  value  of  research  results  is  patent  activity. 
The  number  of  patents  assigned  to  AFOSR-supported  work  would  be  one 
indication  of  the  return  on  Air  Force  basic  research  investments.  When  good 
products,  processes,  algorithms,  or  other  ideas  emerge  from 
AFOSR-sponsored  research,  their  value  to  the  nation  can  also  be  protected 
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by  obtaining  patent  coverage.  While  AFOSR  does  not  appear  to  be  neglecting 
patent  activity,  it  collects  no  data  to  help  ensure  that  the  nation’s  economic 
interests  are  adequately  protected. 


Finding 

The  lack  of  an  organized  database  on  publications  in  refereed  journals, 
citations,  and  patent  activity  resulting  from  AFOSR-sponsored  research  is  a 
serious  shortcoming  for  Air  Force  basic  research  management.  The  absence 
of  such  data  makes  it  hard  to  assess  the  impacts  of  specific  grants  or  to 
compare  the  impacts  of  intra-  and  extramural  research.  Such  data  could  also 
be  used  as  supplementary  information  (along  with  the  earlier-described  quality 
measures  for  research  persoimel),  to  improve  the  pool  of  AFOSR  researchers. 


Technology  Transition 

Another  dimension  of  research  results  is  the  effectiveness  of  technology 
transition  from  basic  research  to  applied  science  and  engineering.  In  the 
current  Air  Force  structure,  the  mission  of  technology  transition  is  shared  by 
AFOSR  and  the  Air  Force  laboratories. 

Leadership  is  clearly  aware  of  the  importance  of  technology  transition, 
and  many  Air  Force  activities  are  designed  to  effect  or  enhance  the  transition 
of  basic  research  results.  These  activities  include  the  intramural  basic  research 
program  itself,  the  spring  and  fall  reviews,  the  maintenance  of  overseas  offices 
in  Europe  and  Japan,  and  the  recent  establishment  of  AFOSR’s  S«feT 
Coordinator  Program.  In  this  program,  some  program  managers  are  selected 
to  serve  as  liaisons  between  discipline-oriented  basic  research  and  the 
multidisciplinary,  technology-oriented  work  of  the  laboratories.  The  designated 
S&T  coordinators  serve  as  resources,  advisors,  and  organizers  of  activities 
(e.g.,  meetings  and  workshops)  for  about  20  major  technology  areas. 

All  these  efforts  are  certainly  merited,  and  there  is  much  anecdotal 
information  to  suggest  that  technology  transition  does  occur  successfully  within 
the  Air  Force.  At  the  same  time,  there  is  once  again  no  organized,  accessible, 
central  information  base  on  technology  transition  from  Air  Force  research 
projects  to  Air  Force  (or  other  DOD)  development  projects.  The 
subcommittee  therefore  could  not  make  any  quantitative  assessment  of  the 
effectiveness  of  technology  transition  for  AFOSR-sponsored  research. 

Some  general  information  was  provided  by  the  AFOSR  scientific 
directors  who  briefed  the  committee.  They  were  certainly  aware  of  their 
responsibilities  to  help  in  transitioning  their  research  results  into  development 
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projects.  (However,  the  subcommittee  did  not  perceive  this  cognizance  as 
uniform  throughout  AFOSR.) 

Additionally,  a  special  briefing  to  the  committee  covered  "Basic  Research 
Initiatives  to  Battlefield  Capabilities."  The  briefing  provided  an  overview  of 
a  dozen  technology  areas  in  which  the  research  results  from  AFOSR  had 
eventually  been  transferred  through  development  into  fielded  capabilities. 
Assuming  the  accuracy  of  this  historical  analysis,  the  subcommittee  concluded 
that  the  traditional  means  of  technology  transition  used  by  the  Air  Force, 
namely,  highly  decentralized  dependence  on  the  laboratories  and  AFOSR 
research  managers,  sometimes  pays  off.  To  what  degree  it  should  pay  off  and 
to  what  degree  it  is  paying  off  could  not  be  assessed  because  of  the  lack  of 
data. 


Finding 

The  lack  of  a  centrally  maintained  database  on  the  transition  of  AFOSR 
research  results  into  applied  science  and  engineering  impedes  Air  Force 
assessment  of  the  effectiveness  of  this  process. 


Contribution  to  Knowledge  Infrastructure 

Basic  research  contributes  to  the  knowledge  infrastructure  for  Air  Force 
technologies  through  both  successful  and  unsuccessful  projects.  Analyses  of 
both  kinds  of  projects  can  contribute  significantly  to  the  value  derived  from 
research.  A  organized  knowledge  base  on  these  projects  could  enrich  all  of 
the  following  activities: 

•  evaluation  and  selection  of  research  proposals; 

•  advanced  stages  of  research  and  development  where  the  engineering 
of  systems  begins;  and 

•  justification  of  the  basic  research  program  to  those  who  make 
decisions  relating  to  its  resources  and  funding. 

Currently,  there  is  no  systematic  capture,  storage,  and  distribution  of  the 
results  of  Air  Force  basic  research  projects.  Much  documentation  does  exist 
in  the  form  of  miscellaneous  documents— published  papers  and  unpublished 
progress  and  project  reports. 

In  selecting  research  projects  currently,  AFOSR  relies  heavily  on  the 
judgment  of  individual  program  managers.  The  operating  philosophy  is  to 
select  top-quality  people  for  these  positions,  assuming  that  the  quality  of 
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project  selection  and  management  will  follow.  While  the  subcommittee 
generally  supports  this  philosophy,  it  still  believes  that  the  lack  of  an 
organized  knowledge  base  on  the  results  of  prior  AFOSR  research  projects 
hampers  the  ability  of  program  managers  to  carry  out  the  selection  of 
top-quality  programs.  The  ability  of  a  research  manager  to  compare  proposals 
for  research  projects  to  the  well-documented  results  and  knowledge  gained 
from  prior  research  is  invaluable  in  selecting  new  projects. 

Li  advanced  development  projects,  Air  Force  laboratories  undoubtedly 
try  to  capitalize  on  knowledge  gained  from  prior  Air  Force  research,  even 
when  projects  are  conceived  in  response  to  Air  Force  requirements  ("customer 
puU").  It  seems,  however,  that  this  transfer  of  knowledge  is  left  largely  to 
serendipity,  and  that  much  could  therefore  be  gained  from  an  organized 
knowledge  base. 

Moreover,  notwithstanding  the  frustration  of  scientific  managers  and 
researchers  with  the  bureaucracy  of  accountability  by  seemingly 
unknowledgeable  people,  significant  justification  will  continue  to  be  demanded 
for  defense  research  that  uses  public  funds.  The  lack  of  a  systematic 
knowledge  base  derived  fi’om  both  anecdotal  and  quantitative  results  is  a 
serious  shortcoming  from  this  perspective  as  well. 

Finally,  such  a  knowledge  base  would  be  an  invaluable  management  tool 
for  the  entire  basic  research  management  chain  of  the  Air  Force. 


Finding 

The  lack  of  an  organized  knowledge  base  (library  function)  of  "what  was 
learned"  fi’om  Air  Force  basic  research  projects  is  a  serious  shortcoming  in  the 
management  and  use  of  project  results. 


Meeting  Long-Term  Scientific  Work  Force  Needs 

The  quality  of  Air  Force  basic  research  depends  of  necessity  on  the 
quality  of  its  long-term  scientific  and  engineering  work  force.  This  work  force 
includes  not  only  researchers  in  the  Air  Force  laboratories  and  in  universities, 
but  also  the  much  larger  contingent  of  those  employed  by  Air  Force 
contractors  and  related  commercial  establishments  that  carry  out  research, 
development,  testing,  and  evaluation  (RDT&E).  To  guarantee  the  reliable 
delivery  of  technology  over  time,  AFOSR  has  been  active  in  ensuring  that 
related  long-term  scientific  work  force  needs  are  met. 

In  this  area,  the  subcommittee  identified  five  indicators  of  quality  activity: 


84 


MANAGING  AIR  FORCE  BASIC  RESEARCH 


•  funding  of  university  scientific  manpower  to  do  research; 

•  number  of  Ph.D.s,  research  fellows,  and  graduate  students  supported 
by  AFOSR; 

•  in-house  programs  (for  postdoctorate  and  graduate  student 
researchers,  faculty  on  research  sabbaticals,  etc.); 

•  short-term  assignments  of  university  faculty  and  graduate  students  in 
Air  Force  laboratories  and  AFOSR;  and 

•  women  and  minorities  supported  at  universities  through  grants  to  Pis, 
research  fellowships,  and  graduate  assistantships,  and  employed  as 
professional  staff  in  the  Air  Force  laboratories.  Women  and  minorities 
represent  an  ever-increasing  proportion  of  the  U.S.  work  force.  Failure  to 
capitalize  on  this  pool  of  talent  would  constitute  a  lost  opportunity  to  improve 
work  force  quality. 

Reasonably  comprehensive  data  were  provided  to  the  coimnittee  on  these 
criteria. 


Funding  of  University  Research  Personnel 

Several  types  of  Air  Force  basic  research  funding  go  to  support  university 
research  personnel.  The  Defense  Research  Sciences  (DRS)  program,  which 
accounts  for  most  Air  Force  basic  research,  is  funded  at  about  $210  million 
for  fiscal  year  1992  and  is  expected  to  remain  roughly  constant  over  the  next 
few  years.^  About  60  percent  of  DRS  funding  goes  to  researchers  at  colleges 
and  universities,  along  with  about  $20  million  in  additional  funds  (in  1992) 
through  the  University  Research  Initiative  (URI)  program.  Thus,  roughly  $146 
million  of  Air  Force  basic  research  funding  goes  to  support  research  and 
researchers  at  universities. 


Finding 

To  achieve  its  mission,  the  Air  Force  must  sustain  appropriate  leads  in 
basic  research  now  and  in  the  future.  A  sufficient  scientific  work  force  will  be 
needed  to  sustain  these  leads.  Current  levels  of  Air  Force  support  for 
university  research  personnel  appear  appropriate,  based  on  the 
subcommittee’s  informal  judgment.  Nevertheless,  especially  in  the  face  of 


^  Sources  of  figures  this  paragraph  are  Headquarters,  Air  Force  Materiel  Command;  and 
personal  communication,  Sally  Brown,  Executive  Assistant,  Director’s  Office,  Air  Force  Office 
of  Scientific  Research,  November  10,  1992. 
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e3q)ected  budgetary  constraints,  a  good  quantitative  estimate  is  needed  of 
future  Air  Force  scientific  and  engineering  work  force  needs.  The 
subcommittee  notes  that  significant  time  is  required  to  bring  research 
personnel  through  the  educational  pipeline,  and  thus,  to  provide  adequate 
support  of  the  long-term  Air  Force  scientific  work  force,  it  would  be  better  for 
the  Air  Force  to  make  its  estimates  conservative. 


AFOSR  Science  and  Engineering  Education  Programs 

Additionally,  AFOSR  devotes  some  DRS  and  URI  funds  (totaling  over 
$15  million  in  1992)  to  support  special  programs  of  postdoctoral  and  faculty 
exchange,  and  science  and  engineering  education: 

AFOSR  maintains  three  dedicated  programs  of  postdoctoral  and  faculty 
exchange: 

•  USAF  National  Research  Council  Resident  Research  Associateship 
Program; 

•  Summer  Faculty  Research  Program;  and 

•  University  Resident  Research  Program. 

It  also  maintains  four  dedicated  science  and  engineering  education 
programs: 

•  Graduate  Student  Research  Program; 

•  High  School  ^prenticeship  Program; 

•  Laboratory  Graduate  Fellowship  Program;  and 

•  National  Defense  Science  and  Engineering  Graduate  Fellowship 
Program. 

In  1993,  about  225  faculty  and  postdoctoral  students  will  be  supported  by 
the  postdoctoral  and  faculty  exchange  programs,  at  least  150  in  the  summer 
program  and  nearly  75  in  the  other  two  exchange  programs.^  All  but  the  first 
are  in-house  programs.  The  science  and  engineering ,  education  programs 
support  about  125  full-time  graduate  student  fellows,  100  graduate  student 
researchers  in  the  in-house  Graduate  Student  Research  Program  (an  adjunct 
to  the  Summer  Faculty  Research  Program),  and  about  125  students  in  the 
high  school  apprenticeship  program. 


^  Headquarters,  Air  Force  Materiel  Command,  FY93  Research  TechnoloQf  Area  Plan,  op. 
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Special  emphasis  is  placed  in  all  these  programs  on  increasing  the 
number  of  women  and  minorities  in  science  and  engineering.  In  the  National 
Defense  Science  and  Engineering  Graduate  Fellowship  program,  ten  percent 
of  the  awards  are  set  aside  explicitly  for  underrepresented  minorities. 

AFOSR  also  encourages  the  development  of  the  future  science  and 
engineering  work  force  by  means  of  its  other  basic  research  programs.  Project 
proposal  review  criteria  specify  that  more  favorable  consideration  be  given  to 
proposed  projects  that  would  strengthen  the  nation’s  base  of  technically 
trained  personnel  (e.g.,  through  graduate  assistantships). 

All  forms  of  AFOSR  support  for  university  research  personnel  provide 
advantages  beyond  the  maintenance  and  development  of  the  science  and 
engineering  work  force.  The  funding  of  university  researchers  greatly  leverages 
Air  Force  resources,  inasmuch  as  university  research  is  funded  by  a  variety  of 
sources.  Additionally,  exchange  and  support  help  facilitate  the  transfer  of 
outside  advances  to  intramural  programs,  and  build  the  university  research 
community’s  interest  in  research  areas  of  special  advantage  to  the  Air  Force. 


Finding 

The  AFOSR  science  and  engineering  education  programs  for  faculty  and 
postdoctoral  and  graduate  students  appear  well  designed  to  meet  their 
objectives.  They  support  a  sizeable  number  of  research  personnel,  and  their 
emphasis  on  in-house  programs  and  women  and  minorities  appears  reason¬ 
able.  Considering  the  future  needs  of  the  Air  Force  and  the  nation  for  a 
healthy  science  and  engineering  work  force,  and  also  the  leveraging  and 
technology  transfer  opportunities  available  through  funding  university 
researchers,  Air  Force  programs  in  this  area  are  highly  valuable.  Their 
funding  levels  should  be  assessed  in  this  light,  and  taking  into  account  the 
results  of  the  earlier  recommended  assessment  of  future  work  force  needs. 


CONCLUSIONS 

The  general  quality  of  research  activities  supported  by  AFOSR  is 
probably  acceptable  and  even  very  good.  But  there  are  significant  deficiencies 
in  Air  Force  procedures  for  measuring  and  assuring  research  quality.  The 
subcommittee  is  convinced  that  the  value  of  Air  Force  intra-  and  extramural 
basic  research  can  be  substantially  enhanced  by  addressing  these  deficiencies. 

1.  Beyond  its  current  review  processes,  AFOSR  should  implement  the 
following  review  procedures: 
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a,  AFOSR  should  randomly  select  projects  representing  perhaps  10 
percent  of  the  annual  Air  Force  basic  research  outlay  each  year  and 
subject  them  to  detailed  external  peer  review.  The  results  of  these 
reviews  should  then  be  compared  with  evaluation  ratings  obtained 
through  internal  review  to  reinforce  the  validity  of  internal  review 
processes.  (The  Department  of  Energy,  for  example,  carries  out  external 
reviews  of  each  of  its  projects  once  every  three  years;  see  Appendix  B.) 

b.  AFOSR  should  conduct  internal,  in-depth  reviews  of  one-third 
of  its  research  projects  each  year,  so  that  every  research  project  with  a 
life  of  three  years  or  more  is  subjected  to  rigorous  benchmarking.  This 
should  be  an  ongoing  process. 

2.  AFOSR  should  institute  a  process  to  monitor  the  quality  of  the  scien¬ 
tists  and  engineers  it  supports.  The  objective  and  subjective  measures 
described  under  the  discussion  of  "Personnel"  above  should  be  applied  and 
updated  annually.  The  following  specific  steps  are  suggested: 

a.  An  organized  database  on  objective  measures  of  the  quality  of 
research  personnel  should  be  created  and  maintained,  to  describe  the 
overall  quality  of  the  pool  of  researchers  supported  by  AFOSR  grants 
and  awards. 

b.  This  database  should  also  be  used  to  improve  the  mix  of 
scientists  supported  by  AFOSR,  without  being  unduly  concerned  about 
the  relative  firactions  of  AFOSR  funds  expended  intra-  and  extramurally. 
Since  the  objective  is  to  provide  the  best  research  for  the  Air  Force’s 
future  technological  needs,  funding  should  follow  the  best  people. 

3.  AFOSR  should  also  institute  a  process  to  create  and  maintain 
databases  on  the  following: 

a.  Publications  in  refereed  journals,  citations,  and  patent  activity 
resulting  from  AFOSR-sponsored  research.  Among  their  other  values, 
these  quality  parameters  can  be  used  to  help  determine  the  relative  value 
and  cost-effectiveness  of  intra-  and  extramural  research. 

b.  Scientific  impact  and  transition  of  research  results  to  applied 
science,  engineering,  and  Air  Force  materiel  acquisitions.  Such  a 
database  would  be  of  great  help  to  AFOSR  research  managers  in 
measuring  the  effectiveness  of  their  programs  and  improving  the  mix  of 
their  portfolios.  This  database  would  also  help  the  AFOSR  Director  in 
prioritizing  areas  of  the  AFOSR  research  portfolio. 

c.  "What  was  learned"  from  Air  Force  basic  research  projects.  This 
information  base  could  help  AFOSR  research  managers  more  effectively 
manage  their  research  projects  and  better  use  their  research  results. 

4.  AFOSR  should  ensure  that  a  good  quantitative  assessment  is  made 
of  the  future  Air  Force  scientific  and  engineering  work  force  needs,  especially 
for  research  and  development.  This  assessment  should  result  in  a  strategic 
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plan  for  the  support  and  development  of  faculty  and  postdoctorate  and 
graduate  students.  Until  this  new  plan  is  in  use,  AFOSR  should  continue  to 
maintain  its  current  funding  levels  for  the  support  of  university  research 
personnel  and  science  and  engineering  education  programs.  (Such  planning 
must  be  based  on  incomplete  facts,  but  the  process  of  planning  forces  a 
valuable  discipline,  yielding  important  insights  and  plans  will  be  amended 
both  with  this  e)q)erience  and  as  more  accurate  information  becomes 
available.) 

5.  The  quality  metrics  identified  by  the  subcommittee  should  be  used 
as  the  basis  of  an  AFOSR  annual  report,  to  describe  the  quality  and  effective¬ 
ness  of  its  program.  AFOSR  should  apply  the  quality  metrics  as  an  ongoing 
audit,  for  better  assessment  of  the  value  of  its  research,  use  of  its  research 
funds,  and  justification  of  its  program.  The  quality  metrics  should  be  refined 
through  their  ongoing  use.  (See  also  the  subcommittee’s  outline  on 
measurement  criteria,  Attachment  C-1.  Other  promising  quality  criteria,  such 
as  the  Malcolm  Baldridge  National  Quality  Award  Criteria,  should  also  be 
considered  in  AFOSR’s  development  of  quality  metrics.) 
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ATTACHMENT  C-1: 

CRITEWA  TO  ASSESS  THE  QUALITY  AND 
RELEVANCE  OF  AFOSR  RESEARCH 

The  subcommittee  that  examined  the  quality  of  Air  Force  basic  research 
began  by  identifying  some  of  the  criteria  through  which  research  quality  could 
be  assessed.  In  developing  its  criteria,  the  subcommittee  considered  relevant 
measurement  dimensions  and  corresponding  metrics;  processes,  means,  and 
tools;  and  implementors  and  executors.  All  these  measurement  features  must 
ultimately  be  well  specified  for  a  good  assessment  of  Air  Force  basic  research. 

A  selected  set  of  the  criteria  presented  below  were  used  by  the 
subcommittee  in  its  analysis.  AFOSR  is  encouraged  to  explore  these  and  other 
quality  criteria  in  developing  good  measures  for  its  research  activities. 


Research  Quality 


Dimensions 

1.  Persoimel 

2.  Research  results 

3.  Contribution  to  the  knowledge  infrastructure 

4.  Assurance  of  long-term  scientific  work  force  needs 

5.  Appropriate  research  management  of  ongoing  activities 

Metrics 

1.  Personnel 

a.  Fellowship  in  professional  societies 

b.  Membership  in  national  academies 

c.  Other  professional  recognition  (prizes  and  awards) 

d.  Breakdown  by  advanced  degrees  and  scientific  and  engineering 
disciplines 

e.  Average  professional  age 

2.  Research  results 

a.  Publications  in  refereed  journals 

b.  Citation  index 

c.  Patent  activity 

3.  Contribution  to  the  knowledge  infrastructure 

a.  Project  documentation  (both  of  projects  that  have  succeeded  and 
of  those  that  have  failed) 
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b.  Project  post  mortems  and  their  documentation 

4.  Assurance  of  long-term  scientific  manpower  needs 

a.  Funding  of  university  research  work  force 

b.  Number  of  faculty,  and  postgraduate  and  graduate  students 
supported  by  AFOSR  (through  research  fellowships,  other  special  science 
and  engineering  education  programs) 

c.  In-house  programs  (support  of  postdoctorate  and  graduate 
students,  faculty  on  research  sabbaticals,  etc.) 

d.  Short-term  assignments  of  university  faculty  and  graduate 
students  in  Air  Force  laboratories  and  AFOSR 

e.  Women  and  minorities  supported  at  universities  through  grants 
to  Pis,  research  fellowships,  and  graduate  assistantships,  and  employed 
as  professional  staff  in  the  Air  Force  laboratories 

5.  Appropriate  research  management  of  ongoing  activities 

a.  Effectiveness  of  interim  measurements  of  project  progress 


Processes,  Means,  and  Tools 

1.  Leveraging  the  research  activities  and  results  of  others,  including  Air 
Force  laboratories,  universities,  other  service  organizations  (especially  the 
Army  Research  Organization  and  Office  of  Naval  Research),  other  federal 
agencies  (e.g.,  the  National  Aeronautics  and  Space  Administration,  Federal 
Aviation  Administration,  and  Department  of  Commerce),  and  industry 

a.  Does  the  planning  and  review  of  AFOSR  research  content 
reflect  appropriate  cooperative  efforts  (forums)  at  the  scientific  level? 

b.  Does  Air  Force  basic  research  adequately  exploit  joint 
projects— at  the  right  level  and  through  appropriate  means? 

c.  Through  what  types  of  efforts  does  AFOSR  interact  with 
industry,  particularly  in  Independent  Research  and  Development  (IR&D) 
projects?  Are  there  barriers  to  these  kinds  of  interaction? 

2.  In-house  research 

a.  Annual  reviews  (Is  balanced  attention  given  to  programmatics 
and  scientific  content?) 

b.  External  review  teams  and  visiting  committees  (What  processes, 
cr  ieria,  and  standards  are  used?  Who  participates?  Are  there  standing 
visiting  committees?) 

c.  Ongoing  internal  review/milestones 

3.  Extramural  (especially  university-based)  research 

a.  Frequency  and  quality  of  reports 

b.  Frequency  of  site  visits  by  AFOSR  program  managers  and  level 
of  interaction  during  site  visits 
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c.  Review  and  decision  process  for  proposals  (both  new  and 
renewal) 

d.  Use  of  forums  involving  principal  investigators  and  peers 


Research  Relevance 

With  regard  to  measuring  research  relevance,  attention  should  be  given 
again  to  all  features  of  the  measurement  system:  the  dimensions  of 
measurement  and  specific  metrics;  processes,  means,  and  tools;  and 
implementors  and  executors. 


Dimensions  and  Metrics 

1.  Use  of  research  results  in  Air  Force  or  other  DOD  systems 

a.  Contributions  to  6.2  and  6.3  activities 

b.  Contributions  to  troubleshooting  of  current  operational  problems 

c.  Contributions  to  meeting  requirements  or  statements  of  need 

2.  Impact  on  other  national  needs 

a.  Contributions  to  U.S.  economic  health  (dual-use  technologies) 


Processes,  Means,  and  Tools 

Use  of  appropriate  evaluators  of  AFOSR  research  relevance  (e.g., 
representatives  of  Air  Force  laboratories;  Air  Force  end  users/operators; 
industry/contractors;  other  DOD  entities,  such  as  the  Army  Research  Office 
and  Office  of  Naval  Research;  or  other  government  entities,  such  as  the 
Department  of  Commerce  or  White  House  Office  of  Science  and  Technology 
PoUcy). 


Appendix  D: 

Meetings  of  the  Committee  on 
Air  Force  Research  Management 


March  8,  1991 

Chairman/Dr.  Schell 

Washington,  DC 

May  1,  1991 

Planning  Session 
(Sponsor) 

Washington,  DC 

June  10,  1991 

Briefings 

AFOSR 

July  16-17,  1991 

Briefings 

(ARO,  ONR,  OSD, 
DARPA,  AF  labs) 

Washington,  DC 

August  15,  1991 

Subcommittee 

Lehmann,  Miley 

Wright-Patterson  AFB 
(Wright  Lab) 

August  20-22, 

1991 

Subcommittee 

Lehmann,  Miley 

Brooks  AFB 
(Armstrong  Lab) 

September  18-19, 
1991 

Briefings  (AFSAB,  NSF) 
Subcommittee  Meetings 

Woods  Hole,  MA 

September  27, 

1991 

Subcommittee 

Rock,  Kerrebrock 

Hanscom  AFB 

November  6,  1991 

Subcommittee 

Parks,  Patel,  Green, 
Decker,  Zraket,  Weiss 

AFOSR 

November  12-14, 
1991 

Subcommittee 

Zraket 

Hanscom  AFB 
(Rome  Lab) 

December  9,  1991 

Subcommittee 

Parks,  Baciocco,  Ippen 

Washington,  DC 
(DOE,  NASA) 
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December  11,  1991 

Subcommittee 

Lehmann 

Brooks  AFB 
(Armstrong  Lab) 

December  12-13, 
1991 

Subcommittee 

Weiss 

Kirtland  AFB 
(Phillips  Lab) 

December  17-19, 
1991 

Subcommittee 

Miley 

Wright-Patterson  AFB 
(Wright  Lab) 

January  29-30, 

1992 

Draft  Report 

Washington,  DC 

February  6,  1992 

Subcommittee 

Miley 

Wright-Patterson  AFB 
(Wright  Lab) 

February  11,  1992 

Subcommittee 
Lehmann,  Miley 
Kerrebrock,  Rock 

Conference  Call 

August  18-19, 

1992 

Draft  Final  Report 

Woods  Hole,  MA 
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